I give up with Test::Aggregate

March 10, 2010 § 3 Comments

I really want to like Test::Aggregate, I really do. But it’s just too damn fragile to get in a decent development environment. I’m really interested in getting our test suite running fast and when Test::Aggregate works, it really does this. Shaving that 1 second off each test adds up pretty fast, and I like that.

However, I do not like this.

That’s the result of the second test in all these aggregates failing, but if I hadn’t told you that, would you have easily known that? The error messages are cryptic as hell. If we do run the second test on its own, TAP does provide the error message (“DBD::Pg::st execute failed: ERROR: column “locale” of relation “artist_alias” does not exist” for the curious).

Until Test::Aggregate can reliably produce these error messages I just can’t bring it into a work environment without risking slowing my colleagues down, which is a real shame as aggregating our tests drops the time down from 248 seconds to just 70. So, the solution?

I see 2 real solutions – the nearest is to use prove -j, which I mentioned earlier, and the second is to help rafl et al work on a forking test harness. I’m really hoping to start work helping in a fortnight. A forking test harness has the benefits of test::aggregate (you can load Catalyst::Test to start a server once, for example), but it doesn’t end up fucking itself quite so hard when things go wrong (the fork might die, at worst). However… (there’s always a but), both of these mean that tests will run in parallel, not in sequence as Test::Aggregate does. The problem with this? Your tests cannot, absolute must, modify the world, and our slow tests are slow partly because they do that – they modify a test database.

What I’d like to press for now, at work at least, is to get all our tests that do not need to test database interactions directly (over half) to talk to mocks, and for our other tests to be run in sequence. Every other test can at least run with prove -j.

Advertisements

§ 3 Responses to I give up with Test::Aggregate

  • Ovid says:

    A couple of quick comments.

    First, Florian has co-maint on Test::Aggregate, so if there are issues, you could talk to him about them. Second, I do realize there are issues with TA, but if nothing shows up in my bug queue (https://rt.cpan.org/Public/Dist/Display.html?Name=Test-Aggregate), I really can’t do much to help.

    I’d be happy to try and help you resolve what’s happening. Can you tell me the version of TA you’re using? Can you provide a minimal test case which replicates this?

    • acidcycles says:

      Hey Ovid, thanks for the reply – and apologies for sounding a bit hostile. This happens with the latest version, and I have no idea how to even approach finding a failing test case here. The project is open source so I can give you a branch where some stuff is failing… but we do have quite a lot of deps (I imagine you have most of them though). Are you around on IRC to chat more?

      • Ovid says:

        No worries about sounding hostile. I’ve also been called on the carpet for that before 🙂

        I’m rarely on IRC, to be honest. Wanna shoot me an email with steps to replicate?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

What’s this?

You are currently reading I give up with Test::Aggregate at Cycles.

meta

%d bloggers like this: