close fullscreen

Ticket

+status
closed
help edit space_dashboard
+priority
low
help edit space_dashboard
+tag
Wagn 1.05+2
Tests
help edit space_dashboard
+commit
 

+issues

help edit space_dashboard

When I've been doing code katas with folks in Chicago, it makes doing TDD in Wagn feel like swimming in molasses.

 

Running a scenario under spork on a new rails project takes just under 3 seconds.   On Wagn, it's closer to 7 for startup plus each step typically takes a second or two, so we often have 20+ second round-trips for a single scenario. 

 

Additionally, our total test suite  ( test:unit + rspec + cucumber ) takes a grand total of about 20 minutes.  This is *waay* too long.   A quick benmark suggests that up to 80% of the time in the unit tests occurs in Card#save.

 

Jon Dahl (some rails developer with a blog :) says:


Most Rails projects I’ve worked on have ended up at around 3,000-15,000 lines of code, with a roughly as many lines of test code, and most have test suites that take a minute or more to run. Our test suite for Tumblon, for instance, churns along for 2.5 minutes. This is a too slow. And slow tests are a problem for at least two reasons: they slow down your development and decrease code quality

 

This is only one datapoint but we're an order of magnitude out of his target.

 

 

 

 

+solution

help edit space_dashboard

In no particular order:

  1. fixture import (aka test db reset) only on request instead of every run  (should save 1-3s? every run)
  2. review all tests for card setup that can be moved to the global fixture set
  3. optimize Card#save   (this is valuable for other reasons as well, obviously)
  4. spot optimize slowest tests.

 

 

help edit space_dashboard

So, we've obviously done a bunch of work on this already.  I suspect that some of the biggest opportunities remaining are:

  1. getting rid of unnecessary time in defaults setting in Card.new.  We could do that by adding :skip_defaults to a gazillion test calls (though surely some of them need the defaults).  I'm guessing much of this will go away when we refactor permissions
  2. further optimizing card#save.  Most of the biggest gains there have been exploited, but I was using a fairly simple case -- there may be other opportunities for optimizing when saving plus cards, cards with many references, etc.
  3. selectively preloading the cache for some specs, but not all.  For the ones where it's unnecessary it slows things down, so it ends up being a wash.  But if we're able to pick and choose, I think we could do some good.
  4. spot optimizing, as suggested above.
So here's the question for the group: do we still want to consider #1 and #2 from the +solutions box above?  I'm not clear on whether those are still consistent with our approach.

I think my new take is that I'd rather spend the time building the new permissions system and then optimizing the tests to work with that.

What tools do we have for measuring performance? Both for tests and for more production like loads.

  --Gerry Gleason.....Thu Jan 06 10:47:46 -0800 2011

not enough. we use new relic, but have only the lowest level of support. I might be willing to invest in more. Lew has a benchmarking spreadsheet somewhere, though I think the old data may not be super useful, as I think it's based on running stuff on his old laptop.

  --Ethan McCutchen.....Thu Jan 06 11:41:08 -0800 2011

fwiw, I lowered this to "low" pri because I think testing performance is more or less under control now, but production performance needs to be a high priority for us. (which I think you were already implying)

  --Ethan McCutchen.....Thu Jan 06 11:42:27 -0800 2011

Yes, exactly. I think it is well under control for testing and that focusing on test performance could have you doing things that are bad for production performance.

When I get the current branch more under control, I may want to try to generate new data and compare to Lew's data for trends and just to be sure nothing has gotten too far out from what it was.

So if you have any useful links on this, please share. Should we have another ticket for production performance, maybe?

  --Gerry Gleason.....Thu Jan 06 13:12:11 -0800 2011

so...close this ticket and add another with the remaining stuff to do and this discussion?

  --John Abbe.....Sun Jan 23 00:40:04 -0800 2011

Sounds right to me.

  --Gerry Gleason.....Sun Jan 23 03:48:02 -0800 2011

further speed up tests

  --John Abbe.....Sun Jan 23 03:55:59 -0800 2011

So, what was actually done here? Just optimizing Card#save ?

  --John Abbe.....Sun Jan 23 04:01:13 -0800 2011

 

+discussed in support tickets

help edit space_dashboard