Software is too expensive to build cheaply...

I saw a presentation today from BEA on the Weblogic Workshop. The tool was pretty nice, but it crystallised an idea that has been kicking around in my head for a while now. Quite simply, I'm not sure these sort of tools are good for development use.

The idea behind the tool is that for every decent software developer out there, there are at least 10, maybe a lot more, who aren't very good. These "developers", often stereotyped as VBA programmers, are people who can do a decent drag-and-drop interface, but can't really build anything complex, either due to lack of native ability or due to lack of training. This was especially true during the dotcom-craze, when millions of people worldwide tried to get into IT when they really didn't have a natural aptitude for it.

(To any VB programmers reading this, I won't apologise. VB has an immense number of followers, many of whom can't code their way out of a wet paper bag. OTH, there are also some very good VB programmers; no stereotype is absolute. There are also a lot of crap Java programmers)

The problem is that there aren't enough good developers out there. The BEA guy gave numbers of about 2 million decent developers vs 15-20 million "HTML hackers", world-wide. I'm not about to question the numbers; certainly, the ratio is either right or optimistic.

It's a well known principle of software development that good developers are considerably more productive than the average, and one of the reasons is that the average is pulled down a lot by the bottom end. The bottom end developers really aren't capable of anything more complex than doing form-based, VB-style of development. In fact, many of them do VB-style of development. However, with MS shifting VB over to VB.NET, many of these people aren't going to be able to make the jump. So BEA came out with the WebLogic Workshop, in the hope of being able to steal some developer marketshare.

All of this is well and good, and to be honest, the Workshop would allow an inexperienced or mediocre programmer to pull together an app quickly. However, these tools don't remove the fundamental complexity of the problems; they only hide it. And that's where the problem occurs.

Take a typical team: 4 to 5 junior programmers, and one senior lead programmer (who is probably with less than 10 years experience, maybe even less than five). The 4-5 junior people are going to be cranking out code, and the senior person is going to spend his or her time reviewing a lot of it to make sure that it's okay and no design flaws have crept in. Now, these 4-5 junior people are keeping the senior person very busy, and the senior developer still needs to crank out the "sensitive" code; the stuff that has to work. Now double the junior developers productivity (or even just increase it by 20%). What happens?

In an ideal world, the team starts becoming more productive. However, the world isn't ideal. In fact, what really happens is that the senior developer can't keep up anymore. The junior staff won't have their work reviewed, or the sensitive stuff slips. Defects start slipping through, hopefully to be caught by a testing team. Most of these defects will be to do with special cases, or with scalability/reliability constraints. In the end, an apparently functional application is built and delivered, but underneath, it's very brittle (just like a lot of VB apps, by coincidence). Quite possibly, the application spends a lot of time in the test cycles, eating up all of those productivity gains. And the junior staff won't be able to fix the bugs, because they won't understand the technology underneath the covers.

The other problem here is that a certain percentage of those junior staff can make the transition to journeyman status; not all, but a reasonable amount. But they won't be able to accomplish it if the real complexity is hidden. They'll never come to grips with the concepts behind the code that make the transition possible. Potential cut short. :(

These tools have a strong place. But I wouldn't want to use them on my mission-critical apps, and not on many of my non-mission-critical apps. If it's a small app, and it will only take a few days, then knock it up with these tools; small investment, small risk. Anything bigger, think twice before doing it like this.

Now, I'm not old enough to remember the debates when 3GL languages were introduced. However, I suspect that the current point-and-click style of tools are at a similar state to the early 3GLs. The first ones introduced more problems than they solved. But they won through in the end, as problems became too big to deal with in assembly code and the 3GLs became more sophisticated. The true crossover point came when it was possible to write a compiler for a language in itself (you can even do this in Java, including writing a JVM in Java (admittedly, it would have to run inside of a JVM, but you could do it)). When these point-and-click tools become sophisticated enough to do that, they'll have made it.

(A secondary problem is that graphical/visual interfaces aren't necessarily any better at organising complexity than normal OO code... But that's another issue, for another day).

That's all for now. Thanks for reading.

For Christmas, I (like many geeks and non-geeks around the world) got a copy of the "The Two Towers" boxed set (the 4 disc one). Should my wife ever read this, thanks again for this present.

In many ways, the essence of my role at work is to foster a learning community. This is more than just a "learning environment" - all a learning environment does is to supply resources for learning. Essentially, it's passive, just like any environment is.

I can't believe I'm writing about this, but here goes...

Offshoring is part of a continuing trend of economic displacement. In its current form, it's been going since about the mid '70s, when cheap bulk transportation (ocean-going cargo ships) combined with an increase in industralisation in undeveloped countries (specifically: Japan and Taiwan at the time) started to result in manufacturing jobs heading to cheaper countries. This trend continued, and now the vast bulk of manufacturing work occurs along the Asian Pacific Rim (though I'd expect Africa to pick up if the political situation there ever stabilises).

"Do The Simplest Thing That Can Possibly Work" - that's what. Now, what does that mean?

I got interviewed a few months ago by a journalist from CIO Magazine. Here's the article.

I'm sick of the term "performance testing". It misses the point.

n a comment on an earlier post, Jon Eaves expressed concern that refactoring is being over-used as a verb. In particular, the line between refactoring and rearchitecting (or rewriting) was being blurred, and refactoring was being used as a label for any activity where you go back and do things right the second time. You know something? Jon's right.

Bucky, who seems to write a bile-ish blog, thinks that test driven development is bad, because he doesn't like compile errors in his code. Because of that, he suggests writing the interface first.

I bought a copy of the famous "Peopleware" book by DeMarco and Lister last week in a second-hand store. I haven't been disappointed in it either. Anyway, in one of the chapters, they cite the infamous Glib's Law:

Anything you need to quantify can be measured in some way that is superior to not measuring it at all.

Some time back, "Jason Yip" came up with a corollary to this which got dubbed (by others) as Yip's Law:

Anything can be made measurable in a way that is inferior to not measuring it at all

This lovely piece of sarcasm is 100% accurate. As DeMarco and Lister points out, Glib's Law does not say that anything can be measured in a way that is cheap, easy, or cost-effective. Typically, for something nebulous like productivity, what you actually end up measuring are second-order effects¹. Now, there isn't anything inherently wrong with this.

DeMarco and Lister make the very strong point that, for this measurements to be effective, they can not be given to management. This sits well with my experience. I, as an individual, can use impartial and incomplete metrics to highlight areas of weakness that I need to improve. Furthermore, I can do this without deliberately skewing the metric; after all, I know that the numbers aren't fundamentally important.

However, the instant these figures get into the hands of management, they become something used to measure my productivity. I will start to have incentives to ensure that the numbers trend in a positive fashion. In other words, in order to get a better performance review (and the subsequent financial rewards), I have a strong interest in optimising the numbers. This invariably has a strong negative effect on productivity (Lines of Code, anyone?).

This is a corollory to Yip's Law that I would like to add now:

Given the choice between not measuring and using an ineffectual measurement, management will always chose the ineffectual measurement, as long as it is cheap

The problem here is that managing by second-order effects is a management anti-pattern. People will optimise the second-order effect at the cost of the primary effect you are really interested in, and this will have serious negative effects. Using the Lines of Code example, developers start cutting-and-pasting large chunks of code rather than simply calling a method; this, in turn, causes defects to reproduce throughout the system², which lowers productivity because of the need to correct this. However, the impact of this on productivity can be hidden simply be cutting-and-pasting more code. :)

----

¹ A second-order effect is a symptom, not a cause. Managing via second-order effects is like placing a person with a fever due to the 'flu into air-conditioning (or an ice-bath) to bring their temperature down.

² I'm really bad for cutting-and-pasting code. Whenever I do it, there will invariably be a syntax error that will prevent the code from compiling. Over the years, I've become convinced that this is a self-defence trick being played by my sub-concious. ;)

Dale Emery proposes a simple test to see if a best practice really is.

I've often encountered the attitude that code or scripts should be commented to the extent that someone unfamiliar with the language being used can understand what's going on. I most recently encountered this with a set of Ant build scripts I had developed.

Doing this would be bad.

Chris Justus managed to totally miss my point. He assumes that I was advocating for crappy uncommented code. That is not the case.

In response to Chris Justus again, here's how I'd comment his example.

Cedric takes exception to my earlier post;; another who didn't get what I was saying. I think that says a lot for my ability to explain ideas, uh?

There's a (somewhat controversial) design principle in object-oriented programming called the Liskov Substitution Principle. One of the classic examples is about Circles being Ellipses.

One of the great strengths of the Java community is the quality of the free (as in both speech and beer) development environments, Eclipse and NetBeans. Although Microsoft let you have the .NET SDK for free (as in beer), there was no free IDE. Until now.

Daniel Steinberg has a note on Java.Net about testing EJBs out of the container, and wonders why people make a fuss about it. There's a simple reason: speed.

Paul Graham wrote about The Python Paradox. Unlike his earlier talk, this one actually had a good point.

A gentle reminder to myself: don't make things more complicated than they need to be.

Esther Derby describes a notice from Gallup about the lack of recognition and appreciation in modern companies.

Dear Mr. Architect:

Please design and build me a house. I am not quite sure of what I need, so you should use your discretion. My house should have somewhere between two and forty-five bedrooms. Just make sure the plans are such that the bedrooms can be easily added or deleted. When you bring the blueprints to me, I will make the final decision of what I want. Also, bring me the cost breakdown for each configuration so that I can arbitrarily pick one.

The New Yorker has an article on talent management in modern corporations. It looks into how companies manage talent to improve performance. Quite a good read.

I'm leaving Suncorp in January, after about 3 years there. I'm taking up a position at wotif.com, where I will be joining a small team that drives their website.

Name: Precision Estimation

AntiPattern Problem

Developers continually get asked to provide an estimate with a high degree of accuracy. They are expected to spend a fixed period of time to produce the estimate.

Andy Marks recently posted a dissection of various categories of build failures. In general, I agree that there are definitely different severities of build failures. The question is: is there a time when a build failure is not important?

I had an interesting conversation with a colleague this afternoon. It centered around what was more important: expressing intent (which I was advocating) vs. removing duplication.

On the XP mailing list, a discussion has been going on recently on how a student at a presentation commented that XP seemed to be fairly heavy. Now, I know that "heavy" and "light" are rather passe terms for describing methodologies these days, but you know, the student was right - for a certain point of view. In the immortal words of Ben Kenobi, "many of the truths that we cling to depend on our point of view."

Using a build server (such as CruiseControl) doesn't mean developers shouldn't run local builds (even though broken builds aren't really as serious as a lot of people make them out to be). So this raises the question: if developers run their build locally, what's the build server for?

A deterministic methodology is one where you lay out all the steps, then following them religiously. An empirical methodology is one where you layout guidelines, and expect people to adapt as circumstances suit.

Christ Stevenson bitched about the Gnome calculator. Apparently, if you enter the equation '2*2+2*2', it gives an answer of 12.

Steve brings up a quote that I've always liked: By definition, risk-takers often fail.

I love it when I get to delete code. Deleting code, particularly dead code, is such a wonderfully therapeutic exercise. You should try it some time.

I had a comment lodged on an older article recently. The poster was complaining about the poor quality of the JWebFit sub-project of JWebUnit. In particular, he was complaining about how it meant their project wasn't delivered on time. There's an anti-pattern here.

Two interesting examples of how the word "accountability" can mean different things.

This has been said before, I know, but it's worth re-iterating: a test should test one thing, and one thing only.

First, some scope definition. Using Kent Beck's terminology, I'm talking about developer tests, not acceptance tests. Also, by one thing, I mean that there should be only one thing that breaks the test (which is very different from saying any failure should only break one test...). In addition, the one thing that breaks should provide diagnostic information - a test failure shouldn't leave you scratching your head to determine the immediate cause

There's always a lot of debate in the various agile groups about what BDUF is, why you should avoid it, when you should avoid it, and why is it bad (or good) for you. I just thought I'd outline my own opinions here.

A very interesting article, originally published in 1992, on Code as Design. Yet more proof that there isn't anything new about Agile (and that's it's best part! ;)

From the JUnit mailing list, courtesy of J.B "JUnit Recipies" Rainsberger

JBR's postulate 1. For every testable design that requires exposing elements "just for testing", there exists an equivalent testable design that does not require exposing elements "just for testing".

From Ron Jeffries, courtesty of the XP Mailing list:

"YAGNI is about coding, not about thinking"

A bloody excellent IT Conversation podcast by Jason Fried of 37signals, taken from O'Reilly ETech 2005.

Jason covers a lot of issues that are at the heart of Agile Development, particularly when it comes to keeping your codebase lean-and-mean, and the YAGNI principle.

Seriously: everyone should listen to this.

Ahhh... that's the first production release of my latest project at work out the door today. I can't talk too much about specifics, but it's not a big secret that Wotif is enabling various B2B aspects of our web site, mainly with the registred hotels. Today saw the first big step in that direction. :) And it feels great to see a new project go out and get used.

Joel's busy complaining that teaching Java in comp-sci courses makes life too easy for people, because they don't have to deal with pointers and recursion. News flash for you, Joel: the times have changed, and new tools are available.

Joel's a pretenious schmuck a lot of the time, but he really does tend to know what he's talking about. His latest article, The Development Abstraction Layer really hits the nail on the head in oh so many way.

Looks like MS is making VS Studio Express free as in beer permanently.

I wrote up my responses to the free beta some time back; like I said then, I think this is a really good move for Microsoft, and it's driven by the quality of the free IDEs for other languages (notably NetBeans and Eclipse for Java).