JUnit Architecture Redux

Looks like I’m talking about another post of Cedric’s It seems he doesn’t like JUnit’s test suites, and decided to write a new tool to solve his problem.

I’m a regular reader of Cedric’s blog, and I’ve been following his periodic posts about JUnit with interest. The more I read, the more I’m convinced that his problems (and the problems of many of the “fellow travellers” who sympathise with him in the comments) is actually because of a misunderstanding of the tool. In particular, the reflection support.

So, I thought I’d write a bit about the architecture of JUnit, and maybe clear up some misunderstandings. Or maybe not; the misunderstanding could always be mine, after all. 😉 Even if it doesn’t, it gives me a chance to ramble on for my own benefit. (In particular, I’m not implying that Cedric doesn’t know all of this already)

At the heart of JUnit is the TestRunner, which comes in multiple forms. They all do pretty much the same thing, however: they take a Test (or collection of Tests), and invoke a method called run(junit.framework.TestResult).

As it turns out, Test is an interface. There are two basic implementations of Test that we need to concern ourselves with: TestSuite and TestCase. (There are a couple of others; we’ll get to those later)

A TestSuite, essentially, is a collection of TestCases. We’ll leave it at that for now. The TestCase is where the action all happens! You may have noticed that the run() method took a parameter, called the TestResult. To make life a bit easier, the TestCase method deals with that for you, ending up inside a method called runBare(). I’ll reproduce the code here:

public void runBare() throws Throwable {
  setUp();
  try {
    runTest();
  }
  finally {
    tearDown();
  }
}

A test will fail if an exception propogates outwards. Exceptions of a particular type (the AssertionFailedError) are recognised as test failures; other types of exceptions will get marked as errors. But I’m getting side tracked. Let’s get back to the runTest() method.

You can define your own runTest() method if you like. The complete code would look a bit like this (but if I’ve got syntax errors, don’t pick on me… this isn’t an IDE):

public class MyTest extends junit.framework.TestCase {
  protected void runTest() {
    assertTrue("This is a useful test".equals(Boolean.TRUE));
  }
}

You’ll need a TestSuite to run that in… (actually, you don’t, but the TestRunners tend to expect TestSuites, for reasons I’ll explain later). Let’s write one of those.

public class MyTestSuite extends junit.framework.TestSuite {
  public MyTestSuite() {
    this.add(new MyTest());
  }
}

Now, if you keep working like this, you’ll end with one TestCase subclass per test you want to write! That would be hideous. So let’s look at what the default implementation of TestCase.runTest() looks like. If you’re not familiar with reflection, don’t worry about it; I’ll explain:

protected void runTest() throws Throwable {
  assertNotNull(fName);
  Method runMethod= null;
  try {
    runMethod= getClass().getMethod(fName, null);
  } catch (NoSuchMethodException e) {
    fail("Method ""+fName+"" not found");
  }
  if (!Modifier.isPublic(runMethod.getModifiers())) {
    fail("Method ""+fName+"" should be public");
  }
  try {
    runMethod.invoke(this, new Class[0]);
  } catch (InvocationTargetException e) {
    e.fillInStackTrace();
  throw e.getTargetException();
  } catch (IllegalAccessException e) {
    e.fillInStackTrace();
    throw e;
  }
}

Okay, so what’s this doing? Very simple. It’s using reflection to call a method in the actual test case object. The method called is the one stored in the fName variable, which (oddly enough) is the test name.

This means we can easily put multiple tests into the one TestCase object, like so:

public class MyTest extends junit.framework.TestCase {
  public MyTest(String testName) {
    super(testName);
  }
  
  public void someTest() {
    assertTrue("This is a useful test".equals(Boolean.TRUE));
  }
  
  public void someOtherTest() {
    assertTrue("This is another useful test".equals(Boolean.TRUE));
  }
}

We’ll need to change the TestSuite for this, however:

public class MyTestSuite extends junit.framework.TestSuite {
  public MyTestSuite() {
    this.add(new MyTest("someTest));
    this.add(new MyTest("someOtherTest));
  }
}

We need a way to run these, however. For now, we’ll use the TextUI TestRunner; it’s dull but simple. We’ll add a main() method to the MyTest class:

public static void main(String[] args) {
  junit.framework.TestSuite testSuite = new MyTestSuite();
  junit.textui.TestRunner.run(testSuite);
}

At this point, the MyTestSuite class isn’t really helping us anymore: we can populate the TestSuite instance inside of the main() method, like so:

public class MyTest extends junit.framework.TestCase {
  public MyTest(String testName) {
    super(testName);
  }
  
  public static void main(String[] args) {
    junit.framework.TestSuite testSuite = new junit.framework.TestSuite();
    testSuite.add(new MyTest("someTest"));
    testSuite.add(new MyTest("someOtherTest"));
    junit.textui.TestRunner.run(testSuite);
  }
  ...
}

And with that, the MyTestSuite class is no more! I hope you weren’t too attached to it.

If we go back to the original runBare() method, you’ll see that it calls two methods: setUp() and tearDown() These are used to arrange for resources to be made available, and to release them afterwards. My simple test case doesn’t need them, so I use the default (empty) implementation in the TestCase class. However, they can be useful: let’s say I’m writing tests that will need a file with certain data; I can use the setUp() method to create the file, and the tearDown() to delete it. This will get run before every test case.

That’s good behaviour if you want to recreate the file every time. Sometimes, however, you don’t want to create it every time; if the contents of the file won’t change, why recreate it all the time? Why not just create it once for the test run, then delete when the test run is over? That’s where the other implementation of the Test interface comes in: the TestDecorator. and (in particular) the sub-class called TestSetup. I can use TestSetup to get setUp() and tearDown() behaviour for the TestSuite, like so:

public class MyTestSetup extends junit.extensions.TestSetup {
  public void setUp() {
    // some code to be called on the setup.
  }
  public void tearDown() {
    // some code to be called on the tearDown.
  }
}

And we use it like this:

public class MyTest extends junit.framework.TestCase {
  public MyTest(String testName) {
    super(testName);
  }
  
  public static void main(String[] args) {
    junit.framework.TestSuite testSuite = new junit.framework.TestSuite();
    testSuite.add(new MyTest("someTest"));
    testSuite.add(new MyTest("someOtherTest"));
    junit.extensions.TestDecorator testSetup = new MyTestSetup(testSuite);
    junit.textui.TestRunner.run(testSetup);
  }
  ...
}

So, as you can see, it’s easy to build a TestCase, with several test methods, aggregate them into a TestSuite, and give the TestSuite some setup and tidy-up code.

At this point, however, we’ve got a bit of a problem. When I write a new test method, it’s not added to my test cases unless I remember to change the main() method to add it by hand. That’s ugly. Fortunately, there’s a nice little convenience method in the TestSuite. To date, we’ve been adding TestCases into the TestSuite. We can, however, get them added semi-magically for us, because the JUnit authors (Beck and Gamma) gave us a convenience method based on reflection. If we add a class object, the TestSuite will automatically create TestInstances for us, one for each method it finds that it thinks is a test. The catch, however, is that the way it thinks a method is a test is that it is public, and starts with the word ‘test’ (Another similar technique is to give the test class into the constructor of the test suite). If you do this, BTW, you don’t need the constructor, as the reflection code calls the setName() method. This makes our TestCase look like this:

public class MyTest extends junit.framework.TestCase {
  public static void main(String[] args) {
    junit.framework.TestSuite testSuite =
        new junit.framework.TestSuite(MyTest.class);
    junit.extensions.TestDecorator testSetup = new MyTestSetup(testSuite);
    junit.textui.TestRunner.run(testSetup);
  }
  
  public void testSomeTest() {
    assertTrue("This is a useful test".equals(Boolean.TRUE));
  }
  
  public void testSomeOtherTest() {
    assertTrue("This is another useful test".equals(Boolean.TRUE));
  }
}

The point about this is that you’ve traded some flexibility (the ability to call the tests whatever you want, plus the ability to control the contents of your TestSuites directly) for a bit of simplicity. This is a 80/20 solution; fortunately, if you’re in a special case where this isn’t any good, you can toss the reflection convenience methods away and do it by hand.

We’re not quite finished here: the main() method we’ve used will happily use the decorator, but the TestCase could be run in other ways (via other TestRunners), and they won’t use the decorator. Again, there’s reflection support for this: if we create a static method called suite(), that returns a Test, it will always be invoked. That makes our code look like this:

public class MyTest extends junit.framework.TestCase {
  public static void main(String[] args) {
    junit.textui.TestRunner.run(MyTest.class);
  }
  
  public static Test suite() {
    junit.framework.TestSuite testSuite =
        new junit.framework.TestSuite(MyTest.class);
    junit.extensions.TestDecorator testSetup = new MyTestSetup(testSuite);
    return testSetup;
  }
  ...
}

Caveat: This only applies to TestRunners; if you chain TestSuites inside of TestSuites, the suite() method won’t be used, and your decorators won’t be created! Still, all good TestRunners should respect the suite() method, and it’s not hard for your custom TestSuites to remember to invoke the suite() method. It would be kind of nice if the constructor that uses the class object could do this, though (and no, I’m not full of ideas on how to avoid the recursion that would cause, right now…)

So why wouldn’t you always use the reflection support? The reflection support provides an 80/20 solution (okay, more like 95/5); sometimes it doesn’t do what you want. I blogged a while back on a data-driven approach to creating unit tests that needs the flexibility that ignoring the reflection support gives.

Oh, it turns out that we can drop the main method; if you need to invoke the class, you can do so on the command line like this:

java junit.textui.TestRunner MyTestCase

or, for a fancier interface:

java junit.swingui.TestRunner MyTestCase

Anyway, that’s the quick overview of the JUnit architecture. What you should have come away with is a reasonable understanding of how the TestSuites, TestCases, and TestDecorators relate to one another, and why the reflection support is the way it is. I was also going to comment on some of the perceived weaknesses of JUnit (especially the ones that Cedric brought up), but well this post is already too long; another time.

For a final point, however: Cedric talked about a prototype tool he had which essentially replaced the reflection support used by JUnit with annotations (ala JSR 175, in Java 1.5). This is all well and good, but if it doesn’t use the annotations as an extra layer, the way JUnit uses reflection as an extra layer, then it will be a step backwards. A fancy step maybe, but still one going the wrong way.