Defect Driven Testing… and You’re the Driver

November 21, 2010 § 3 Comments

Recently at $work I’ve been doing, quite frankly, a poor job of maintaining one of my areas of responsibility. I am the developer of a script that deals with the migration of about 12 million rows of data from one data format to another. This work has become sufficiently complicated that it can’t really be done in SQL, and has a set of Perl classes to do the migration. It’s a very critical piece of work right now, because it defines how long we have down time, and we only get to run this script once – if it’s wrong, then I may have corrupt a decades worth of historic data (effectively an audit trail). This is Not A Good Thing.

As you can see, I’m aware of these problems, and have made progress in increasing the speed of script, along with the quality of it. What I’m not so happy to tell you, is that I’ve also managed to commit this script in various forms of stability – from syntax errors, start up runtime errors, to errors that happen at the end of the script – something that you still end up waiting a good hour until you find out. I’m a perfectionist, and no doubt this cycle of “hack-break-hack-break-hack-hey it works!-hack-break-hack-fix” is getting extremely tedious, and doesn’t feel at all representative of my actual ability. Finally, that little moment of enlightenment has come and it’s time to sort this out. My solution? >buzzword<developer driven defect driven development>/buzzword< (someone please make me a manager).

Defect Driven Development

Before we get on to talking about my revolutionary testing approach which will probably cure world hunger, we need to discuss the concept of Defect Driven Development (DDD, for now). DDD is a testing methodology, where by you ensure you have a test for a bug. Dead simple. Here’s the basic methodology: you work on a great new feature and release it. User U finds a bug in your feature and reports it. You, the developer, write a test and then fix the bug. Now the bug will never occur again (theory here, work with me…)

Now here’s the twist:

You Are the User

While developing, consider yourself the user as well. So you hack away, and something breaks. At this point, you don’t fix the break, you write a test to reproduce it. And I mean everywhere. Even down to the smallest details – no matter where the bug is. Syntax error? Reproduce it in a test1. A function crashes with some input? Reproduce it in a test. The world is beginning to implode? Reprod… ok, it won’t really help here.

I doubt that I’ve really made much of an impact of this post but that’s my point. This is a very simple way to get tests into your codebase, and it has one some excellent consequences when you actually get to the end of your feature. Every new feature or enhancement really needs tests to go with it, but tests are hard, and testing is boring. I’ve written unit tests before that test basic exercise, but finding the edge cases that do break the system is difficult. I don’t think my tests often help me develop my feature, they are more useful later as I start to integrate tests for bug fixes. But with this new approach, you automatically do get some tests of where things have broken in the past – and I believe if it’s broke before, it’ll probably break again, unless you’re taking steps to prevent that.

1 While yes, the code wouldn’t actually run, in Perl we can still have a test to make sure something compiles. use_ok in the Test::More distribution can do this, for example.


§ 3 Responses to Defect Driven Testing… and You’re the Driver

  • Geoff says:

    So, do you ever feel there can be too many tests?

    What happens to code flexibility with all those tests? Who will want to improve the design, with all those tests to migrate to the new design? And more…

    Testing is good, testing is great, writing as much test code as real code, or more, for any reasonably sized project seems like destructive protectionism to me.

    • acidcycles says:

      Well, it really depends what you are testing, I’ve found with this approach you can end up engineering a system that is testable, but you don’t end up testing too much of the internals – which gives you flexibility to rewire and refactor things.

      You shouldn’t really end up with a lot of these tests; the intention is to add a test for something that has already broke. While yes, you may have to maintain tests a little, the test should be fairly minimal to reproduce the problem.

      However, I will point out I’ve only started using this methodology recently, and not religiously. Also, I work on a very small team, and the person who maintains the test is normally the same as the one who writes them. There is a possibility this doesn’t scale to larger teams.

      Thanks for your comment!

  • Jacques Chester says:

    Reminiscent of the Personal Software Process in terms of the “all mistakes are defects” approach and attention to eliminating them in future.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

What’s this?

You are currently reading Defect Driven Testing… and You’re the Driver at Cycles.


%d bloggers like this: