It was a while back when I attended the Developer Testing Forum. The were many good presentations about what developer testing can do for us. It was closed with a captivating presentation by Kent Back. They were all good. Yet I left somewhat unsatisfied.

For one thing, most of us are already developer testing enthusiasts. It was really preaching to the converted. Most topics are rather uncontroversial. We all want more unit testing. Even pointy haired boss would say so.

The real question is, why doesn't everybody do it?

Here are a few common objections and my responses.

1. We have no time - that's sad, and it will probably be paid later.
2. Our software is monolithic and tightly coupled, making it unsuitable for unit testing - we know this is no way to build software, loosely couple is the way to go
3. Unit testing is difficult

Unit testing is difficult? As a developer testing advocate, I maintain it is simple and beneficial. On the other hand there is also some difficulties I do not want to downplay. Perhaps this is the right time for me to lay them out.

1. It is hard to write program that can be controlled by another program.

It is hard enough to find programmers who can write good program that functions according to a spec. I think it is a lot harder to expect programmer to write program that be also be controlled by another program, which is what unit testing requires.

There is one class of program that can be easily controlled by another program, that is utilities. This is not surprising because by definition utilities is a small piece of code to be called by another program to carry out some functions. There is also little surprise that many literatures focus on testing utilities, which is the low hanging fruit in unit testing.

Unfortunately most of the code written are not utilities. They interact with users, read and write to the database and the disk, control the flow of other processes. They react to and change system states. Testing these code requires design and instrumentation and demand more sophistication from programmers.

2. Techniques are in its infancy and are not well publicized

Since utilities are easy to test, one thing to do is to identify logic embedded in main body of code that is better isolated into utility functions. Queuing, sorting, searching and various algorithm are everywhere. It will a big win to isolate them just by the virtue that a feature can be tested much more easily.

The point I want to make is there are techniques like above for better unit testing. But they are often in its infancy and are not well publicized. Just like good programming, good unit testing is not easy to come by.

3. Extra level of abstraction can complicates design

Since we cannot always replicate the exact context for invoking the tested code, we often parameterize the code so that we can pass in a mocked environment. For example fetch_record() may means fetch from the database in the module. But since we rather not use the database in unit testing we parameterize it to take an extra resources parameter. In runtime context it will be the database. But in unit test context it will be an artificial data source.

In some case we find the added layer of abstraction actually improve the code because it is no longer hard coded to work in a single context. On the other hand flexibility come at a cost, no matter how small it may be. Having a few configuration variables and the complexity starts to multiply.

While complex and monolithic code is obviously undesirable, how is highly flexible, highly configurable code compares to a piece small and well integrated code? When simplicity is a virtue, flexibility can be a liability.

4. Test to death

Unit testing is a key tool in agile programming. You can be bold in refactoring when there is a suite of unit tests to back you up. Ironically extensive set of unit testing can also become the cement that harden your code and make changes difficult. If you diligently add tests for every subprocess of your code, chances are a lot of them will break when you change the code. It doesn't necessary mean your code has problem, it only means you have to maintain the test code and adapt it to the new version. It is arguably more difficult than changing to code itself because first you have to distinguish between a real problem from symptoms than merely need up keeping. On one hand unit testing give you the confidence to refactor. On the other hand too detailed testing can bog you down.

Finding the right level of test that cover the function but not so rigid that it breaks easily is also an art to master.

I have cited these few problems I have experienced not to suggest developer testing is not valuable. Instead I want to bring up a discussion that there are skills and technique required for successful testing. Also many issues can be mitigated by better tools and better programming language like AOP.

2006.04.15 [software] - comments