I give up with Test::Aggregate
March 10, 2010 § 3 Comments
I really want to like Test::Aggregate, I really do. But it’s just too damn fragile to get in a decent development environment. I’m really interested in getting our test suite running fast and when Test::Aggregate works, it really does this. Shaving that 1 second off each test adds up pretty fast, and I like that.
However, I do not like this.
That’s the result of the second test in all these aggregates failing, but if I hadn’t told you that, would you have easily known that? The error messages are cryptic as hell. If we do run the second test on its own, TAP does provide the error message (“DBD::Pg::st execute failed: ERROR: column “locale” of relation “artist_alias” does not exist” for the curious).
Until Test::Aggregate can reliably produce these error messages I just can’t bring it into a work environment without risking slowing my colleagues down, which is a real shame as aggregating our tests drops the time down from 248 seconds to just 70. So, the solution?
I see 2 real solutions – the nearest is to use prove -j, which I mentioned earlier, and the second is to help rafl et al work on a forking test harness. I’m really hoping to start work helping in a fortnight. A forking test harness has the benefits of test::aggregate (you can load Catalyst::Test to start a server once, for example), but it doesn’t end up fucking itself quite so hard when things go wrong (the fork might die, at worst). However… (there’s always a but), both of these mean that tests will run in parallel, not in sequence as Test::Aggregate does. The problem with this? Your tests cannot, absolute must, modify the world, and our slow tests are slow partly because they do that – they modify a test database.
What I’d like to press for now, at work at least, is to get all our tests that do not need to test database interactions directly (over half) to talk to mocks, and for our other tests to be run in sequence. Every other test can at least run with prove -j.