Michael Schubart’s Weblog

The Banana Holder.

Itoki Building, Chuo-ku, Tokyo.

Here’s a neat idea: Jester is a tool that randomly introduces bugs to your production code (a copy of the code, of course). Then it runs your unit tests to see if they fail. They should - and if they don’t, you might want to add some more unit tests in order to get better coverage.

This is an alternative to using a code coverage tool in order to test your tests. In fact, it might even be more effective than code coverage because code coverage only tells you whether a line of code has been executed or not. It does not help you understand the quality of the tests. Here’s an example:

Let’s say we have the following production code:

public static int maximum(int x, int y) {
    if (x > y) {
        return x;
    } else {
        return x; // Bug: Should return y.
    }
}

And we have the following “test”:

public void testMaximumA() {
    MyMath.maximum(4, 3);
}

If we run this test using a code coverage tool, the tool will tell us: “50% coverage”. Not too bad, but the test did not catch the bug. Hm, maybe 50% is just not enough. Looking at the output of the coverage tool, we see that the “else” statement is never executed. That makes sense: We don’t have a test where the y parameter is the maximum. Let’s add this case:

public void testMaximumB() {
    MyMath.maximum(4, 3);
    MyMath.maximum(10, 22);
}

Now the code coverage tool tells us that we have achieved 100% code coverage. Perfect! But the test still does not catch the bug. Let’s use Jester. Jester randomly introduces a bug. For example, it might turn the “greater than” into a “less than”:

public static int maximum(int x, int y) {
    if (x < y) { //Bug introduced by Jester. Was: (x > y)
        return x;
    } else {
        return x; // Bug: Should return y.
    }
}

When we next run the unit tests, they will still pass. And Jester will say: “Hey, I just broke your code. How come your unit tests still pass? The test of the tests has failed!” So Jester has found a problem that the code coverage tool could not detect. Note that we did not (directly) find the bug in the production code. We only found an issue with the tests. Once we improve the tests, they will hopefully reveal the actual bug. Let’s do that, just to be complete:

public void testMaximumB() {
    assertEquals(4, MyMath.maximum(4, 3));
    assertEquals(22, MyMath.maximum(10, 22));
}

A very original and elegant concept!

I found Jester via IBM developerWorks - in spite of their awful RSS feeds…