What I don’t like about TestNG

Seeing all the comments today about using annotations to declare test cases reminded me that I never got around to writing about what I didn’t like about TestNG

I wrote quite a long post a couple of months back about the architecture of JUnit, explaining a lot of the complications. Nothing in that article was particularly world-shaking, and it was information I picked up easily enough by reading the JUnit source code.

That article was meant to set up the context for a later post, which I never made. So here it is. 🙂

Ogres are like onions

One of the real strengths of JUnit is the fact that it’s been built up in a number of architectural layers. When one doesn’t fit, you can always peel it back and get to a lower level. Being able to use a data file to create tests is a very powerful technique, and very easy, and I just don’t see how that’s done under TestNG^[1]. That’s just one example of how diving under the standard layer is useful.

In order for a JUnit replacement to gain traction, it will need to have similar layers, to allow developers to get past the normal 80% level. The annotation support is an excellent way to build test cases up, I admit, but it must not be the only way.

(Of course, the architectural layering of JUnit is one of the reasons Cedric doesn’t like it. 😉

Does this look like a nail to you?

JUnit’s got huge traction in the Java community because of the tool support. Being able to fire up a unit test in your IDE, and jump straight to the line where an assertion fails, is great. Being able to trigger the tests with a key stroke is wonderful as well; less context switching. The emotional impact of the green/red bar can’t be overstated either. 🙂 A new test harness will need at least some of this level of tool integration to be widely accepted (all of it won’t come until after the acceptance is already won).

If the only way you can easily launch tests remains from the command-line, it’s not going to work very well. Also, it’s annoying that you can’t launch all of your tests, each with their own set of properties. Case in point: how can I easily launch, in TestNG, both my older JUnit tests and the new-fangled ones?

Log this!

I hate the way the only log format for TestNG is a HTML file. HTML files are great for viewing “as is”, but not for transforming to other formats. The XML log file format supplied by the ant <junit> task is used extensively despite its flaws, because it is good for transformation.

I’m not saying that TestNG should use the same log file format, but it should at least have a way to produce a format aimed at translation, not just viewing. A viewable format is only an XSL transformation away, after all…

Reset to zero

I don’t like the fact that the one test instance is reused between test methods, even in the JUnit “compatibility” mode. Okay, again that’s a deliberate choice by Cedric, who doesn’t like the “instance-per-test” nature of JUnit, but the compatibility mode shouldn’t change the behaviour.

Not everyone bothers to use tearDown() to clean up properly, or setUp() to initialise. This could lead to a lot of broken test behaviour. Here’s an example:

public class ExampleListTest extends TestCase() {
private List test = new ArrayList();

public void testAddItem() {
test.add(new Integer(1));
assertEquals(1, test.size());
}

public void testEmptyList() {
assertTrue(test.isEmpty());
}
}

This test will fail under TestNG, in the compatibility mode. With annotations used instead, it will still fail. This rather simple example will not work in a shared test instance environment, without a setUp() and tearDown() method^[2].

Having an instance per test is a good thing, not a bad thing, because it increases the level of isolation. Sure, it can be annoying to set up data more than once, but that’s what the TestSuite was for. What was needed was a better way to define TestSuites, not dropping the instance per test (and thus just having TestSuites).

Where’s Thom Filicia^[3] when you need him?

There’s not a single Decorator in sight in TestNG. TestDecorators are a very powerful feature JUnit, particularly when combined with TestSuites. One example is using a TestSetup decorator to give you a setUp()/tearDown() for a suite (or suite of suites).

Nowhere in the current implementation of TestNG can I see a way to do this. Cedric, we need a way to assign setUp and tearDown methods to a group, not just a test class or method. Of course, there’s no centralised definition of a group in TestNG, so that might just be a tad hard to do.

Why would you want this? Consider a test group that executes against a remote server. It would be really good if, before the test group starts, you can check to see that the server is up and ready for you. Afterwards, you may want to shut the server down, or restart it.

Back to my monkey habits

Not to be nitpicky, but… I absolutely loathe the use of the ‘I’ to mark an interface. Can’t stand it. Makes me shudder and have flashbacks to when I used to write Microsoft COM code in C.

In conclusion, I’d just like to say

Okay, absolutely none of that is meant to be an attack on Cedric. I’ve never met the guy, but I’ve liked a lot of what I’ve seen him write^[4], and I like the fact he’s willing to put his stuff out there for people to see and use.

Nor is this meant to be an attack on the tool: it’s still immature and evolving. Sure, JUnit was largely written on a plane on a trip between the US and Switzerland^[5], but the supporting infrastructure around it has evolved over several years. I don’t expect that to pop up over night for a new tool.

What it was, I hope, is constructive criticism of a promising new idea. I’d love it if Cedric addresses some of these issues in TestNG, but I won’t cry if he doesn’t; after all, at least some of the points I raised are deliberately done the way they were.

It’s also meant to re-iterate a point I made in my last post: I really believe that the best way to get some of these features is to do them over the JUnit source. There’s only a couple of key classes that need to be altered, and JUnit will suddenly be annotation-enabled. With some luck, it will probably work with most of the various tools around as well.

Heck, this might even get me motivated to do it myself. 🙂

—-

[1] Okay, I can see one way: create a new ITestMethodFinder instance, and then hack the SingleTestRunner class to support a Factory pattern based of the configuration properties instead of just having two hard-coded choices.

[2] Simon Harris made a good argument recently for using tearDown() methods more. Still, a lot of people don’t really bother; after all, most of us probably don’t have 2000+ test cases with lots of data that we need to worry about.

[3] The interior designer in Queer Eye for the Straight Guy.

[4] Of course, he’s been known to fail to get what I write about in the past. 😉 But then, I’m not always as clear as I should be in my writings.

[5]. Largely true, but remember the original design work was done on the SUnit tool for Smalltalk; what was really written in a plane trip was the port to Java of an existing design. And the nice GUIs weren’t there then either.