I really, really, shouldn’t bite, but… Cedric is asking about dependent test methods and if they are evil Well, they may not be evil, but they are at least mildly naughty.
There are reasons why JUnit has separate test instances per test method. The major reason is independence; you can (or should be able to) run the tests in any order. Tests should set themselves up as needed, and tear themselves down as needed – you shouldn’t write tests assuming that previous tests have been called. People still do, of course, and you get odd bugs, but at least it’s _their_ fault, and not the fault of the framework they are using.
In JUnit, at least, the order that tests are run is decided by reflection – the tests are executed in the order they are declared in the class file. This, in turn, is dependant on the compiler: the Sun javac will list methods in the order declared in the source, while the IBM jikes@ does it in reverse order. I have a vague recollection that @gcj does it alphabetically, though I’m probably remembering a stupid joke. The point here is that the order of methods in a class file is something not under your control. Requiring your tests to rely on an ordering not under your control would be silly.
(TestNG, by contrast, runs tests within a test class in the order that the annotations are discovered – again, something not under control)
This has some interesting consequences. For example, in IntelliJ, I can run individual test methods. How annoying would it be to go into a test, run it, and see it fail when it passes when I run the entire test class? Or have it pass when it failed in the larger environment? These wouldn’t be good, right? In the default GUI for JUnit, I can re-run individual tests. How bad would it be if they started passing because a test that ran after the first unsuccessful run changed the world to allow it to pass? There are also test runners that run tests in different orders – for example, the Continuous Testing Eclipse Plugin can prioritise tests for earlier running (notably the most recently failing tests). Using that feature with dependent tests is just asking for randomised failures!
What really surprises me is the context that Cedric is asking the question in. He’s adding a rerun failing tests” mode (and appears to be claiming credit for the concept… naughty Cedric!). He notes that he needs to have the dependant tests run first, and he has to include them in the specially generated testng-failed.xml file. TestNG, of course, gives you a way to “mark tests as dependent which is fine and dandy for the ones you _know_ are dependent. Of course, because TestNG uses the same instance to rerun tests, every test method in a test class is potentially dependent on the others, through their common instance variables. So this new “failing-test” feature of TestNG will merely result in having randomised failures due to dependencies and side effects that the developer was not aware of!
So to sum up: dependencies that you know about aren’t that big a deal. Dependencies you don’t know about are. When you rely on implicit orderings, you often introduce unexpected dependencies.
Share this:
Software is too expensive to build cheaply.... 
From what I’ve seen (admittedly not much), TestNG is aimed at integration and system testing, not (just) unit testing. People abuse JUnit as a general test framework that it was never intended to be. (Similar story to Ant.)
I’d argue that with integration/system testing the tradeoffs are different. You’re dealing with dependencies anyway and system state may be expensive to build (especially for large and distributed systems).
(In part, I think Cedric is talking about this type of scenario because he mentions running thousands of tests overnight. That does not sound like the typical sip-a-coffee, watch-the-green-bar usage of JUnit.)
JUnit is a generalised test framework. You can use it to run nice fast unit tests or long running acceptance tests. The only “unit” focus of JUnit is the name.
For the fast cycle that TDD requires, you require fast tests. That is not to say that every JUnit test you write needs to be fast – what it means is that you need a way to identify the subset of tests that give you confidence for TDD, and make them fast so that you can run them in a TDD cycle.
Actually, to be honest, I don’t mind the _explict_ dependencies that Cedric has introduced in TestNG. Marking dependent methods explicitly is good. IMHO, finding ways to make the dependency part of the test is better. 🙂 All Cedric has done, IMHO, is created a way to declare a test out of several steps.
The _implicit_ dependencies created by having a single test instance, OTH, are a problem.
You can use JUnit for all kinds of testing. However, there are deficiencies. It doesn’t naturally support groups of tests with dependencies and common setup/teardown; it doesn’t naturally support multiple threads. Those are unremarkable issues for a unit testing framework. Arguably a class and its methods should be testable in isolation. It makes me think that the deficiencies are deliberate, and that they’re the result of focussing on doing one thing well.
BTW, I agree that implicit, unintended dependencies are a bad thing. I don’t want to detract from the main point of your post at all.