Sick of “Performance Testing”

I’m sick of the term “performance testing”. It misses the point.

A test is a pass/fail situation. A performance test says something like “Can this webapp handle 200 concurrent users”. You get a Yes or a No. And in many projects, it’s something you do late, with the usual grief when the answer comes up (nearly invariably) as No.

A far better attitude is to think in terms of “performance profiling”. Rather than ask “Can this webapp handle 200 concurrent users”, ask instead “What does 200 concurrent users, on this configuration, look like?”. And ask “What does 100 users, and 300 users, on this configuration look like”, and “What does 100, 200 and 300 users on this other configuration look like?”

If you build a performance profile of a system, you will understand what it can evolve towards. Knowing the answer to “Can this webapp handle 200 concurrent users?” doesn’t help much if you are faced with a need to grow (later) to 400 users. Profiling, however, would.

Now, most “performance tests” actually do aim to build a profile. But the problem is that this profile is usually gathered at the end of the process of building an application. You don’t have time to experiment really; in fact, any experimentation is probably only done if the answer is “No”, and it will be rushed. In any case, you won’t really end up with a comprehensive profile, because you’ll only really be caring about the expectations of the performance tests devised earlier.

The real problem is that performance test scenarios are rarely realistic. Users don’t act in nice predictable ways that a test robot can simulate in large numbers. Users act like real people. If all you have are performance tests, you’ll be hard pressed to explain why the server starts off working well in the morning, but grinds to a halt in the late afternoon when the (smaller) evening shift takes over. A performance test isn’t likely to see what happens when people don’t log off, but let their sessions expire instead. But a performance profile will let you match the poor performance with a corresponding profile, and maybe figure out that the problem is the users not logging out properly, with the resulting memory being used in session state.

When an engineer designs an engine, they don’t just see if it can generate 1000 horsepower they get a picture of how it behaves in a wide variety of conditions. There’s no reason not to do this in software as well.

If you’re trying to go for a component- or service-oriented architecture, it’s even more important. Not understanding the profiles of the various pieces in the architecture will give you a mish-mash of a system; just like if you took that 1000 horsepower engine and attached it the transmission from a clapped-out Ford.

Anyway, that’s enough for this particular discoherent rant…

Author: Robert Watkins

My name is Robert Watkins. I am a software developer and have been for over 20 years now. I currently work for people, but my opinions here are in no way endorsed by them (which is cool; their opinions aren’t endorsed by me either). My main professional interests are in Java development, using Agile methods, with a historical focus on building web based applications. I’m also a Mac-fan and love my iPhone, which I’m currently learning how to code for. I live and work in Brisbane, Australia, but I grew up in the Northern Territory, and still find Brisbane too cold (after 22 years here). I’m married, with two children and one cat. My politics are socialist in tendency, my religious affiliation is atheist (aka “none of the above”), my attitude is condescending and my moral standing is lying down. View all posts by Robert Watkins