This is obvious, but differences between environments cause problems. You can expect bugs to cluster around them.
Case in point: on my current project, we use a hand-maintained schema for production (and manual testing), but our unit tests go against one that is generated from the Hibernate schema. Naturally, we are seeing bugs related to differences between these, the most recent one being about the width of particular columns.
Any time you accept a difference between your development environment and your production environment, you are introducing a vector for bugs. If you use a cluster in prod, but not in development, expect bugs about clustering. If you use a different database (or schema ;), expect database related bugs. If you typically run your production app for a month without restarting, but your development environment only goes for an hour, expect bugs related to endurance.
This is not to say that having differences isn’t the right thing to do. It can be, especially if the difference is expensive to fix (e.g. giving each developer a $250,000 server cluster is probably not viable). However, you need to take care around it.
Differences in environments is one of the best arguments for a build machine (particularly an automated one). It is generally viable to have at least one environment that is “prod like”; use this for your automated tests.
Share this:
Software is too expensive to build cheaply.... 
A build server may not be the best for trying to catch those “long running” bugs – but I guess there is no way to do that easily – unless you have run a release candidate for a month prior to going to prod (not sure if that is always feasable).
You might be surprised, Michael… a build server can do 8 hour endurance runs over night, and 60 hour runs over the weekend. This should suffice to show up anything obvious.
No substitute for a month of endurance testing, of course, but still better than nothing.
I was just thinking about this exact scenario a few days ago. We’re about to deploy a big update and I’m not sure our production database is identical to our test database. It seems like it wouldn’t be that hard to write a tool that compared two databases and produced a diff report, showing exactly which tables, views, columns, keys are different between the two.
Yes Robert, you could actually give it a hammering in a short amount of time – in most cases that is probably “enough” diligence (I guess you can aim to process the same number of transactions).
I once had a problem that took about 1 to 2 months of heavy prod use to surface. Turned out it was a dodgy SMTP server that would go into “drip feed” mode and eat up my web threads (so I moved to JMS, and also overwrode the default infinite timeout on the JavaMail API !! whoops).
And, lets be honest, who hasn’t at one time or another done some “clever” multithreaded code, and got tripped up by a deadlock that only happens on a 4 way prod machine (when you only have a 2 way or a single processor dev machine !!). I stick my hand up. I ignored all the advice – threads are too easy in java (and too tempting), and too easy to get wrong (note: this was in early J2EE days, and I had no other options at the time).