It's no fun to write about stuff that everyone agrees on, so I'm going to write about something that almost everyone seems to disagree with me about:

100% code coverage is almost always possible. It's a valuable target, and it's a worthwhile goal in and of itself.

The primary argument that gets dragged out repeatedly is that 100% code coverage doesn't indicate anything about the quality of your tests whatsoever. It's certainly possible to get code 100% covered without writing a single assert, or by writing incorrect or incomplete tests. If you don't actually assert everything you care about properly, code coverage doesn't seem to be very valuable at all.

While there is a lot of truth to this, unless you have dishonest developers, that code coverage percent does indicate an attempt at quality tests. And really, an honest attempt is all we can ever ask for.

I'm willing to hypothesize though, for the sake of argument that somehow the class under test somehow is 100% covered by wrong tests, incomplete tests, or tests with no asserts whatsoever.  Let's call it an Exercise Suite instead, because that's all it does; it exercises the code.

I'm going to still contend that that coverage metric is a quality indicator. The important thing to notice is that it means you've somehow managed to exercise 100% of your codebase.  This is actually a huge accomplishment that means a few things:

(1) You don't have dead code in your class-under-test.

If you can't exercise every branch in a given class, you've got dead code. Dead code should always be removed to keep from cluttering up future maintenance efforts. (Think you might need it again? That's what version control systems are for -- You can always go back and get it again.)

(2) Your codebase can be loaded into a test harness.

This is actually one of the hardest parts about testing codebases written in procedural languages, or languages that allow scripts, or "includes".  The contents of a given file can't be brought under test reliably and repeatably unless it is possible to load it into the test harness without it actually doing anything until it's exercised by the tests. If you've ever tried to get a legacy PHP project under test, you'll know what I'm talking about here.

(3) You've figured out how to avoid/mitigate side effects.

Code that accesses a database, the filesystem, a web service, hardware, etc is very hard to exercise quickly and without side effects. If your exercise suite has managed to get 100% coverage, while still running fast, reliably and repeatably, you've scored a major victory.

(4) You can detect a myriad of run-time errors.

"Failures" are generally what you have when a test fails. You care about "errors" too though: Errors are all the things that can go wrong at run-time that are obvious just by executing the code with certain inputs. Exercise alone can uncover a number of these. We routinely catch null pointer exceptions in our test suite (attempts to access properties that aren't correctly initialized yet), not through failed tests, but instead simply through exercise. The test runner catches many of them, and always shoves those directly in our face to be solved.

I think my unpopular opinion comes from two things about me that puts me in a bit of a minority as far as developers go:

(1) I've spent a bunch of time actually doing TDD, and I expect unit-tests to look like they were written before the code even if they weren't. If you're writing tests before code, as TDD prescribes, you're generally not going to end up with uncovered code. At any point in the process, the only asymmetry that's likely is that there are tests that are trying to test code that doesn't yet exist. To have uncovered code in TDD generally means you've done something wrong.

(2) I'm extremely accustomed to working with dynamically-typed languages, so I've learned to rely on unit-tests for almost every aspect of verification. I use them to debug. I use them for syntax-validation. I use them to ensure I'm interacting with other objects in ways that they expect (ie, "Did I get that parameter order right in that method call?"). There is no static type-system, so I don't have any quality safe-guard other than the tests I write. In that scenario, uncovered code is a risk I don't want to take -- it could even contain syntax errors!

In short, the hypothetical Exercise Suite has a lot of value in and of itself.

I just finished reading an article on the The Maintainability of Unit Tests and a lot of what was being said there is contrary to my experiences with unit-testing, so I wanted to suggest a few counterpoints.

Basically there's a discussion of how unit-testing can hinder productivity when you have to make major changes to code under test, and more than a few commenters have questioned a goal of 100% coverage.  I'd like to go through those points one-by-one, as well as through some additional points I hear quite often.

"External functional tests would allow more maintenance flexibility under the covers while still finding regressions."

Here's a quote I love to drag around:

“In the industry, people often go back and forth about whether particular tests are unit tests. Is a test really a unit test if it uses another production class? I go back to the two qualities: Does the test run fast? Can it help us localize errors quickly?” (Michael Feathers,  Working Effectively with Legacy Code -- a really great book in my opinion!)

So, unit-tests are simply tests that run fast and help you quickly pinpoint where an error originates from.

External functional tests, on the other hand, are neither fast nor useful at telling you where the problem occurred.  The end result is that you can't run your external functional test suite as you work in small iterations improving your codebase.  You can't run it after every compile and before every check-in, because it takes too long!  And when a functional test suite finds a bug, it can certainly tell you a bug exists, but it's completely useless and showing you where that bug occurred.

"Major changes to a heavily unit-tested portion of code can lead to tedious test changes that take up as much as 85% of the effort"

There are two different scenarios where I've seen this statement applied:

For new code

Instead of doing Test-first development, or test-driven design, our team often postpones writing tests until (1) the requirements have been completely fleshed-out and (2) the approach is known to be a good one.  I think that's perfectly fine -- we're prototyping.

Ideally a prototype would be largely discarded when it's done and TDD would begin for the production-quality code (It would probably be pretty wasteful to not copy/paste large chunks from the prototype code though).  It's critically important that you don't try to prototype your way to production-quality code, so it's best to stop prototyping as soon as the unclearness of the task becomes clear.  After the point that the requirements and the approach are clear, any programming that you do without tests will just be the kind of cowboy-coding you were hoping to avoid.

For code already in production

Any major change should be executed as a series of minor changes.  Every minor change should be preceded by introducing a failing test.  If you follow those two rules, you predictably and systematically meet your goals while guaranteeing that you don't introduce regressions.

It's worth it to take the time to write the kind of code that doesn't need to be revisited over and over in the future, because it was done properly the first time while you had the best understanding of it.

Count yourself lucky that you had all of these existing tests that probably just require minor tweaks.  Take note of how little manual testing you had to do, and how few regressions you introduced.

"It's impossible and fruitless to unit-test view components / UI controls."

It's certainly not impossible.  In our actionscript / flex code, we routinely get 100% coverage on UI controls.  UI controls inevitably have logic and state, and it's not really something we try to avoid.  A checkbox for instance has a state (checked or not) and it has presentation logic to decide whether to show the little black "X" or not.  To move that logic and state somewhere else would be silly and to throw up our hands and say "We can't unit-test UI controls!" would be untrue.  It's true that there are important (visual) functions of that control that still fall outside the realm of unit-testability, but that doesn't mean you can't get that code 100% covered by tests.

"Some code is so trivial that there's no point testing it."

One example given to illustrate this point was a DTO with 20+ properties on it.  What's the point of testing that those properties can be written and read?  It's so trivial that it can't be wrong!

Actually that's not true.  For example: properties always have initial values, and it's easy for a piece of code to rely on a specific default initial value that could be changed by future maintenance.  That's a breakage that static analysis (the compiler) isn't going to know about.  Sure this kind of bug is unlikely to occur in any particular commit, but over time it becomes more and more likely, and your unit tests will always be there to prevent it.

It doesn't matter how trivial a piece of code is: humans can always find ways to get a bug into it.

It's also still very easy to get 100% coverage on that class without writing tests for it at all.  It can be tested indirectly by getting all classes that use it 100% under test.  If you can't get it 100% under test by getting its consumers under test, then it likely has unnecessary features or dead code.  Clean that unnecessary maintenance overhead out and you'll get that 100%.  Is it still a unit-test if you don't have that class directly covered?  Don't worry about that -- ask yourself if that class is covered directly well enough for you to find defects in it quickly.  If it's still not, then we're likely not talking about a trivial piece of code (like a DTO) at all.

"There's a law of diminishing returns as you write more and more minor tests to get 100% coverage."

In practice, we actually see the reverse.  The devil does appear to be in the details:  those things that we thought were minor understandably didn't get much thought during implementation either.  We often don't find bugs in our code (if we're doing post-implementation unit-testing of legacy code) until we actually get down to those minor details.  We're routinely surprised at the issues that we find in the last 20% of our coverage pursuit.

"Unit-testing is not a guarantee of correctness, so we shouldn't get too extreme about it."

I don't want to get too far into explaining the problems with this, but let me just propose the following:

"Ropes can sometimes break and snap, so mountain climbers shouldn't be too strict about using them."

It's absolutely true that unit-testing (even with 100% coverage) does not guarantee correctness.  Anyone who's ever done any significant unit-testing has to wonder at some point "Who tests the test?".  A unit-test is simply a double-check.  If you have bad tests, passing them doesn't mean anything.  Unit-testing is not a silver bullet that will solve all your quality problems: it's just an extremely valuable tool.

Habit is habit, and not to be flung out of the window by any man, but coaxed down-stairs a step at a time.
-- Mark Twain, "Pudd'nhead Wilson's Calendar

In my last entry I talked about how difficult it is to transition a team from being a deadline-oriented team to a quality-oriented team.  The summary: It's extremely difficult to change to a quality-centric approach, simply because it's a massive cultural change.  It requires the formation of entirely new habits, and the casting-away of old ones.  Many developers will resist this change in every way, because it's just completely foreign.

I wrote about 12 stages of unit-testing consciousness as well:

1. run existing tests
2. maintain existing tests
3. understand the importance of code coverage
4. write new tests for code that is written by someone else who understands testability
5. understand the problems of dependencies and what constitutes testability
6. write code and tests for scenarios with no heavy dependencies
7. understand the role of mocking / faking
8. write code and tests for scenarios with heavy dependencies by mocking / faking
9. understand the limitations of coverage analysis
10. understand the value of writing tests before code
11. refactor legacy code to be testable in ways that aren't necessarily safe
12. refactor legacy code to be testable in ways that are safe

In order to get a developer to that 12th stage of consciousness, that developer has to go through each stage, one at a time;  Some of them can be pretty tricky to pass through too.  Let's talk about how to do that as gently as possible.

1. Running Existing Tests

In order to get to stage 1, the team needs to know how to run the existing tests.  Of course, it's essential to first have existing tests.  This is where we get our first prerequisite: The Testing Guru.

The Testing Guru

The Testing Guru is someone who can preach the word of TDD, and knows how to answer the hard questions that will come along the way.  This person will have to be comfortable setting up the initial framework and teaching the others on the team how to get through each and every one of these stages.  Quite frankly, I think that if you don't have a developer that is already knowledgeable in the area of TDD, you're probably not going to succeed at getting anyone on the team to stage 1 or beyond.  It's just too tough of a switch without good solid guidance, and occasional hand-holding.

Continuous Testing

Assuming you've got a testing guru, and that person has started writing some tests, everything is so much easier if you're committed to automated continuous integration (CI) and have a continuous integration server that can run the tests for you.  This is a huge help because instead of trying to start a new policy/habit where people run all the tests, you can have the CI server do it for you, and have it report a build breakage whenever there's a failure.

If you're doing continuous automated testing, eventually people will start modifying code in ways that requires the tests to change.  Before the team gets to that point, the guru should let them know how to run the tests themselves, and inform them not to commit any code if there's a failing test (because that will be considered a broken build).  Stage 1 has basically been reached when everyone is running the tests often enough to not break the build.

In my experience, if the tests are not run as part of the CI server's work, they'll simply be neglected over time, and the testing guru will repeatedly find broken tests that he'll need to fix, because no one else will be bought-in to the process (yet).

2. Maintaining Existing Tests

At some point before the entire team is comfortable with running the tests, some tests will get broken and they'll be blocking a developer from committing his work.  When these come up, the guru should be ready to do some pair programming and show the developer around the test structure and how to fix the broken test.  Enough sessions like this will lead to developers who can fix simple broken tests on their own.

This is a bit of a large step.  As developers maintain tests, they learn to write tests.  They'll start to see what more complex tests look like.  They'll start to understand the fundamentals of testability.  This stage could take some time for the entire team to achieve.

Information Radiator #1:  Test Count

An information radiator is basically a succinct and impact-full visual representation of some information, preferably in a central public place where everyone can see it.  It could be as simple as a graph that you print-out and pin to the wall.  The information that we'd like to show is the trend of the change in the number of tests.  At this point, the CI server should be reporting on the number of tests that are in your test suite.  The team should commit to making sure that that number doesn't regress.  If you set up an information radiator to track the test count over time, when someone solves a broken test by taking the easy route and removing it instead of fixing it, it will be an obvious dip in the line chart.  More visualizations like this will be necessary in the next stage too...

3. Understanding the importance of Code Coverage

This is another stage where the CI server plays a critical role: You'll need to get code coverage reporting integrated with the rest of the build process.

Information Radiator #2: Code Coverage %

The reports should be as easily available as possible to everyone.  This should probably be via a website (so developers can drill in to see branch-by-branch coverage metrics), as well as an information radiator.  Most developers will genuinely be interested in this stat, so if you can make it obvious and public they'll pay attention to it, and be more apt to want to contribute to improving it.  It's important too to track the change-trend of coverage over time, so developers can start to understand what factors effect it and how.

Once they see the holes in coverage in the current tests, they'll be interested in plugging those holes.  They'll see that coverage itself is a good metric for measuring how well testing is going.  Putting a simple metric to the concept like that can have a crystallizing effect.

Developers might want to run the coverage tools locally as well to help them find holes in their testing before they commit.  The guru should be ready to help with that.

4. Writing new tests for code that is written by someone else who understands testability

As the team sees that code coverage number fluctuate, they're incentivized to begin writing their own new tests.  This is where our team introduced the one-unit-test-per-code-commit rule.  It basically goes like this:

There must be at least one new unit-test in every code commit (No excuses).

There might be some developers on the team who have managed to avoid fixing existing tests while other developers have progressed onto this stage.  This one-unit-test-per-code-commit rule will bring those developers out of the wood-work, so that they can be brought gently to this step.

There will be times that a code change is so insignificant that a new test doesn't make sense in that area of the code base.  That's okay: in order to satisfy this rule, developers should be free to find the easiest piece of code anywhere in the codebase to write the most trivial testcase they can.  It doesn't matter at all if it's not a high-value test!  Keep that in mind.  This is supposed to be a gentle introduction.  The guru should be available to help developers find those dead-simple pieces of code (and possibly demo how the coverage tools can be useful for identifying them).

The entire team should be looking for the one test in each peer code review as well, so they hold each other responsible.

The guru should be ready to help developers that want to test some more difficult piece of code.  He should pair program with any developers to help them advance as quickly as possible through the concepts associated with writing tests (There's no reason to have the entire team advance at the pace of the lowest common denominator).

Over time, the test count trend will slope upward, and you might even see a positive trend in the code coverage percent (though don't expect it!).  The team will start to get more comfortable with writing tests and using the coverage tools (to find the easy ones).  Eventually the code with easy-test potential will start to be harder and hard to find as those specimens all get under test.  This will push developers to try writing tests for code that is increasingly difficult.

5. Understanding the problems of dependencies and what constitutes testability

As developers work through writing their tests for the 1-test rule, they'll start to realize that dependencies and coupling in general make testing hard.  There will be parts of the codebase that they won't know how to test, and they'll start to see the patterns in why that code is so hard to test.

This experience will lead them to an understanding of what constitutes testability.

6. Writing Code and Tests for Scenarios with No Heavy Dependencies

When developers reach stage 5, they'll start to write their new code in ways that are more testable (ie sprout classes and methods, as defined in Working Effectively with Legacy Code).  They'll start to ask questions about how to write code that is more testable.  They'll start to ask how the more difficult concepts could possibly be tested.

With the understanding from stage 5, they'll be ready to start writing code and tests for scenarios that don't have heavy dependencies (like the database!).  The guru should be available to help them through those concepts, probably through pair-programming.

Developers at this stage will start to feel a much larger degree of confidence with code that is more heavily tested, whether it's their own code, or they're maintaining code written by another developer.  They'll also start to see how much nicer their code is to maintain when it's been written with testing in mind.

7. Understanding the role of mocking / faking

By the time developers are writing their own tests, they have probably already seen (and maybe maintained) some tests written by the guru that use mocks and fakes.  They'll see how mocks can swap out their more tricky dependencies so they can have fast tests that run without a lot of overhead, and so they concentrate on the code-under-test.

They'll get comfortable with the mocking framework and be ready for the next step...

8. Writing code and tests for scenarios with heavy dependencies by mocking / faking

This step is pretty much self-explanatory, but it's another big one.  Developers that reach this stage will still be thinking of testing as the second stage in the process, but they'll be able to struggle through writing testable code and the tests for it in new-code scenarios.  They'll also be able to factor some code out of the existing codebase and bring that code under test.

At this point the team should commit to ensuring that the coverage percent should not regress.  It might even be useful to have the CI server report a broken build in the case of this regression.  That way everyone has the same interests in understanding the concept of code coverage and the tools for evaluating it.  The team certainly has all the skills at this point to ensure their own success.

The coverage percent trend should only go up at this point, but that will get more and more difficult, pushing the developers into subsequent stages.

9. Understanding the limitations of coverage analysis

Along the way, some developers might notice that they get bugs logged against code that they have 100% under test.  This is normal of course, because 100% code coverage doesn't guarantee correctness, it just guarantees that the code is being 100% exercised, and the code is 100% testable.  While those are extremely useful metrics for determining how well a set of tests cover the behaviour that's expected, developers should understand that there still might be holes in their tests.  Any bugs that are found in 100% covered code will indicate the need for still another test.  Fortunately if that portion of the code is 100% under-test, it will be easy to add a new test.

Information Radiator #3: bug count

Alright, you're probably already keeping track of this, but it's really useful at this stage to show the trending of the bug-count and how it corresponds to the trends in number of tests, and code coverage.

10. Understanding the value of writing tests before code

Many developers won't ever get 100% into test-first development, but they'll all realize that code that is written without testing in mind is often really hard to test.  Tests must at least be considered before (and while) the code is being written.  The guru might help illustrate this concept with the use of ping pong pair programming.  It's basically TDD in a pair programming scenario with a bit of a fun competitive twist.

11. Refactoring legacy code to be testable in ways that aren't necessarily safe

Developers will naturally start to refactor existing code so that it's testable.  They'll more-than-likely do this in a way that is not very safe at all given the chicken-egg scenario where you shouldn't refactor code that isn't already under test, and you can't easily get some code under test until it's refactored.

As the team digs further and further into the legacy code to bring it under test, they will start to see that they're causing new bugs in the process.  Ideally, the guru should try to help the team avoid this stage entirely, but I included it because it seems like every team goes through it, and struggles within it.  The bug count metric might provide some impetus for getting into the next stage.

12. Refactoring legacy code to be testable in ways that are safe

The guru should show all the weird tricks (ie "seams" in Working Effectively with Legacy Code) to get some previously untestable code into a test harness somehow so that it can first be tested before it's even modified.

The team should commit to not making changes without first having the code under test.

It's not until this point that the team is experienced enough to be able to set code coverage goals for itself and commit to a schedule of coverage increases.  The coverage trend will be invaluable in setting realistic goals.

In Closing, Some Caveats:

• In my experience, a team is unlikely to complete all 12 stages in a matter of weeks.  It might be possible in a matter of months, but that would be pushing it.
• The quality of estimates for how long work will take will suffer as developers get used to how much time it takes at first, and later how much time it saves.
• Any return to deadline-driven development, no matter how brief, completely disrupts and undermines the process.

How we got here

Product owners and stake holders have a fixed set of features that they need implemented, and so management then has the choice to put a focus on either quality or speed-of-delivery.  Historically that focus has overwhelmingly been on speed-of-delivery, and so the deadline dictates how much quality a team has time for.  Very few programmers have never worked within the confines of a deadline.

Deadlines have little to no regard for quality.  Quality is sacrificed at the altar of the deadline in order to deliver the necessary features at each stage as fast as possible.  Few people involved in the decision-making process are unaware of the sacrifice that they're making, but it's a trade-off that is often considered acceptable anyway.

In general, management that makes that choice for speed-of-delivery over and over again is unaware of the various studies of the effect of quality in the long run.  The basic principle they're missing is that software that has fewer defects is easier to maintain, so its continued development is much faster than software with more defects.  So quality leads to higher speed-of-delivery in the long run .  To choose speed-of-delivery over and over again instead of focusing on quality is like playing a chess match and considering only one move ahead.  But this is not a natural or intuitive concept to grasp!  Management and developers alike understandably question how we can possibly be faster over the long term by spending so much time on quality.

So that's the way it is in many software development shops, and that's the way it always has been.  Developers, being the clever bunch that they are, optimize their workflows and tools to deliver software as rapidly as possible.  The desire for faster speed-of-delivery at the expense of all else has been culturally ingrained in the brain of every developer I've ever met.

Understanding How Deep the Problem Is

It makes sense then that people are not ready to shift the focus to being quality-centric.  Not only is the justification non-intuitive, but they're simply not equipped in any way to meet quality requirements.  They lack the tools, the training, and the experience.  It's even been found that given a task with *no deadline*, developers will tend to invent their own deadline, just because that's how they're comfortable working.  It's simply not enough for management to direct the team to be quality-focused.  The change in focus to quality-centric goals should be expected to be just as difficult a transition as many developers had transitioning from procedural to object-oriented programming (if not more).  We're talking about undoing a lifetime of work habits and learning completely new ones.

The New Toolset / Mindset

The primary solution we've got for really being able to work with a quality mindset is TDD.  It comes with a bunch of standard tools (unit-testing frameworks, mocking libraries, code coverage analysis tools, inversion of control containers, etc.) and a bunch of clear cut rules on how to measure and achieve an acceptable level of quality.  And TDD has really proven itself to those that have made the transition:  IMVU and flickr, for instance, can deploy new code to their customers many times in a single day because they can rely on the quality brought to them by TDD.

But developers that are new to the concepts don't have that experience of success, and they don't know any of the tools, and they don't have a clear picture of how to achieve their quality goals.  These factors contribute to the steep learning curve of TDD and the quality mindset in general.

The Biggest Obstacle of All

The biggest obstacle of all is that there's a chicken/egg scenario where developers are naturally trying to learn about unit-testing by writing tests for their existing codebase, and it just so happens that existing code is the hardest kind of code to write tests for.  They're simply not ready for the work at that level of difficulty.

There are roughly a dozen stages that any new developer needs to go through in order to be able to work on a team that's doing TDD:

Stages of Unit-testing Consciousness:

1. run existing tests
2. maintain existing tests
3. understand the importance of code coverage
4. write new tests for code that is written by someone else who understands testability
5. understand the problems of dependencies and what constitutes testability
6. write code and tests for scenarios with no heavy dependencies
7. understand the role of mocking / faking
8. write code and tests for scenarios with heavy dependencies by mocking / faking
9. understand the limitations of coverage analysis
10. understand the value of writing tests before code
11. refactor legacy code to be testable in ways that aren't necessarily safe
12. refactor legacy code to be testable in ways that are safe

Each of these steps requires new skills, new tools, new concepts, and new psychological buy-in.  To expect a developer to be able to write tests for his existing codebase as a first step, is to expect a developer to skip all the way to stage 12.  It's simply not realistic.

Quality is REALLY Slow in the Short Term

Even if a developer knows the methodology and toolset inside-out, there's a shocking realization that in the short-term, quality really slows down the development process.  The SQLite project for instance boasts over 600 times the lines of code in their tests than in the product itself .  For a TDD-neophyte, this looks like an unrealistic and wasteful amount of work, but the SQLite developers know exactly how quickly and easily they will be able to make future changes and still maintain quality.  That's an understanding that can only come through experience though.

It's Faster to do a Manual Test than to Write an Automated Test

While absolutely true, this is another product of the short-term thinking mindset that developers have had hammered into their minds for decades.  Obviously if a manual test takes 1 minute to run and a similar automated test takes an hour to write, there's a specific number of iterations after which the the automated test is time well spent (60 iterations to be precise!).  In a normal development day it's common for me to run my automated tests 10 times per hour or more, so I recoup my upfront test-writing time very quickly.  Again, it's difficult to see this, or even see why I'd even want to run my tests so often unless you've had the experience.

The First Step is Admitting We Have a Problem

There's a deep cultural change that needs to happen in a person to make them understand, and it's not a quick or simple process by any means.  There are real impediments to switching our course towards quality that we need to first recognize so that we can overcome them.  A quality focus is such a huge shift in paradigm that we can't expect our teammates to simply buy right into the concepts that help us achieve quality, or even the rationale.  Once we've admitted that we have a problem, we can start to talk about ways to most effectively convince our teammates about the value of the focus on quality and the value of TDD in achieving the goals of that focus (I'll talk about those in the next post).

If we were building cars, we would probably first prototype a body design with CGI to be able to collect feedback before building the car itself. It's much cheaper (effort and materials wise) to iteratively develop the body with a 3d model than with the actual materials that would be used.

If we were designing websites, we might prototype the design in photoshop for a bunch of iterations until the client is happy. The great thing about that is that the web designer can actually chop up that protoype in photoshop, and use it for images as part of the work after the client has approved the design stage. In this case, the prototype isn't a throw-away prototype -- it's actually useful to bring into the implementation stage.

In software development we can (and usually do) prototype similarly. There's usually a point in the design process where a developer needs to "try some stuff out and see what works". Once the developer has proven a methodology with code, the prototyping phase can mostly be called complete. Throwing away that code would widely be considered a huge waste though, so it usually ends up being used in the final implementation as well.

Prototyping VS. Hacking

It's probably safe to say that many developers do prototyping (a design methodology) and implementation at the same time bouncing back and forth between the two as necessary, blurring that line between design and implementation.

This way of developing is seductive, because you get straight to banging on the keyboard, which feels like you're making progress, but it's really upstream in the design stage where you get the opportunity to save the most amount of time. If you don't distinguish between design and implementation, it's tough to tell which stage you're in at any given time, and so it's tough to determine if you're in the right stage.

You can imagine how wasteful this back-and-forth would be if you were an automative engineer instead, and you swapped back and forth between prototyping and working on the implementation details. As you iterate on the prototype, those details you worked on in the last iteration could be completely removed -- that's a big waste of time, and it's often emotionally difficult to let go of design decisions that have undergone a lot of implementation work. So while realistically design never really stops, you do want to do as much up front as possible.

So there are definitely right and wrong ways to prototype, but it's extremely valuable and worthwhile if done correctly. I doubt many developers would argue that sometimes you just need to try stuff out and see what works. Very few developers are interested in development methodologies that omit the ability to prototype.

So What about TDD?

This seems to me to be the most often used argument against Test Driven Development (TDD). I agree that TDD doesn't work well for tasks that don't have clear-cut solutions, but TDD is still always applicable downstream after the solution has become more clear. The trick is to reach a certain level of solidity with your solution's plan *before* you get into TDD.

Here's a case in point as Ron Jeffries attempts to "solve sudoku" with TDD without even knowing the rules of the game. If you follow his method of solving the problem and compare it against Peter Norvig's design-first approach, TDD starts to look like a miserable failure. Indeed, some have dragged this out as a case against the utility of TDD "in the real world".  Ron Jeffries' example doesn't really make the case against TDD at all though; It simply shows that you have no business being in the TDD stage before you fully understand the problem and you've done adequate design. TDD doesn't preclude other design methods (like prototyping) at all.

And in cases of large architectural issues, prototyping is likely not the first tool that you should turn to -- it's too detail oriented, and you can iterate much faster on designs expressed via whiteboard doodling, flowcharts, UML, etc. You might move to prototyping at some stage later though before you completely decide on some particular design. The important part to keep in mind though is that you're not in the implementation stage -- you're planning.  Changes are cheap and easy at this stage, and it saves you so much more time than if you just jumped in and started developing.

The software that comes out of the prototyping process is a nice side effect, but it's almost never ready for production. This is where TDD comes in. If you have a clear understanding of the solution you want to implement (whether you used prototyping or not) then your design is sufficiently complete to start TDD. In this way TDD and prototyping are not opposed at all. Prototyping is just one possible methodology in the design stage that should precede implementation. TDD just says that tests should be written before implementation. It doesn't say anything about what else you do prior to implementation.

So, there's nothing un-TDD about prototyping a solution, setting it aside, writing your tests, then writing an implementation that passes those tests (based on the knowledge you got from your prototype). Ideally you're working with units that are single-minded and small so copying/pasting from the prototype isn't even necessary, but to be honest, I think some copy/paste from the prototype is fine. If it's done in a careful way where you're not hurting the quality of the final code (because prototypes generally don't care all that much about code quality), you might as well make use of your previous work, if you can. You just have to be careful that it's the quality you want, but then again TDD will force a lot of the quality.

The Steel Thread

The concept of the steel thread is one that I use quite often on the team that I work in when we're working on complex tasks. It's definitely a loose sort of prototype, and it's meant to prove the basics of our core concepts. Basically, we concentrate on the core functionality of the complex task, putting priority on the completion of the "happy path", the path of execution without error scenarios, special cases, etc. Once that core functionality is in place, the general design can be considered verified, the necessary components can be easily identified, and the existing work can be evolved to something high-quality (possibly through a fresh start with TDD). Work is easily divided amoung teammates after this steal thread has been acheived.

Spike Solutions

Spike Solutions are like the opposite of the steel thread in many ways. Where the steal thread tries to prove the viability of the entire solution (in a breadth-first sort-of-way), a spike solution will try to prove the viability of a part of the solution that isn't very well known, in a depth first sort-of-way. It's useful when there's a particular piece of the solution that we're not sure will work, so we prototype that part as quickly as possible to make sure our overall solution won't fail because of that single part.

The point of the concepts of steel threads and spikes is to tackle the unknowns early on and prove the viability of the entire solution as early as possible. Once everyone is relatively sure that the team is on the right path and the design is firmed up, you're ready to get into the more detail-oriented aspects of development.   It's not until this point that I'd switch to TDD, and start writing my tests for the implementation stage of the work.

Is this just BDUF?

I don't think so.   I'm not proposing a huge documentation process that specifies every aspect of the final product.  I'm simply talking about spending some time to loosely figure out what we're going to do before we do it.    It's time well spent, because it's just so much cheaper to make major changes at this earlier stage in the process.  We'll still figure out the minor details as we go.

"We will encourage you to develop the three great virtues of a programmer: laziness, impatience, and hubris." -- Larry Wall

In this quote Larry Wall makes some great points about what it means to be a great programmer in a zen-like,  toungue-in-cheek sort-of-way.  Let's look at one aspect of the Virtue of Laziness.

Every reasonable developer knows that he should write his code to be easily maintained.  As part of that work, it's natural for that developer to want to try to make his code as ready for future change as possible.  Taking that desire a few steps further, he might want to write code that solves future needs in addition to the current requirements, so that that code doesn't even need to be revisited.  Unfortunately, that's a common mistake that leads to a lot of wasted time and less maintainable code.

I'd like to propose a principle of software development that I've come to believe:

The most maintainable code is the simplest possible code that meets all the requirements.

This is almost a tautology because simple code is (by definition) easier to understand, so it's easier to change/enhance/extend.  Code that does more than what's currently necessary is not as simple as it could be though, so it's not as maintainable given the current requirements.

One might argue that code that does more than what's necessary can meet future requirements without change, thereby proving itself to be more easily maintained (no maintenance required!), but all of that hinges upon the developer's ability to consistently and accurately foresee future requirements.  Otherwise that developer has:

• wasted time developing unnecessary functionality
• decreased the maintainability of the codebase

Even when everybody is pretty certain of the future requirements...

Now let's consider the case where the developer is almost absolutely sure that the requirements will change a certain way, and so the code should be written to handle that scenario as well.  Wouldn't it be better to do that work now?  No.  It still takes more time and effort (at a time when we know its not necessary), and it's still less maintainable code until those requirements change (in case other unrelated changes are required).  On top of all that, there's *still* risk that the requirements won't change in the anticipated way!

Even when everyone is absolutely 100% certain of the future requirements...

Now let's consider the case where everyone is 100% certain that the requirements will change in a specific way, and you can write your code to handle that way as well as the current way.  In that infrequent scenario, you don't face a bunch of losing propositions, so it does make a bit more sense.  However, I'd still like to pose a question:  how much more work is it to do the additional work later instead of now?  Truthfully, the amount of effort is virtually identical regardless of whether you do it now or later.  That means you can enjoy maintaining the simpler version until the time comes that it needs to be changed to meet new requirements.

This isn't a new idea I've dreamt up: it's the YAGNI principle, and it's an essential part of Uncle Bob's 3 Rules of TDD.  It's also a great way to avoid Purple Code.

(It's important to note too that YAGNI doesn't say to avoid making your work as futureproof as possble.  YAGNI just says not to put extra code into making that possible.  For example, if you create a class for a UI button that will be used to save some user data in a form, don't call it "UserDataSaveButton" -- just call it "Button".  It's no extra work and the re-usability possibilities have increased dramatically.)

We're not engineers building bridges;  we don't work with unchangeable steel and concrete, and we can accomodate changing requirements easily.  It's unrealistic and wasteful to guess that we know what future requirements will come, especially when the potential work saved in the long run is neglible.

Development Time VS. Maintenance Time

There's no doubt that the original development stage of a piece of software can be time-consuming.  It's interesting though, when you consider the difference in time spent in development vs. time spent in maintenance.  At what point do you cross into maintenance mode?

Let's face it: you're actually also in maintenance mode pretty much right away, unless you're some superhuman that conceives of the whole system at once and doesn't make any design tweaks (or even typos) while coding.  But you're not.  So the development-time VS. maintenance-time dichotomy is a false one.  You're always in maintenance mode.

So if you spend so much time (all your time) in maintenance mode, then it makes sense to do every little change and tweak in such a way that your subsequent tweaks and changes are as easy as possible.  We should be spending the majority of our energy optimizing our time spent on maintenance and not our time spent on development.

Now it's natural that one might say "well why not optimize both?", and that's a valid question.  There's no good reason not to try to optimize both, but we have to remember to never ever ever optimize the development process at the expense of the maintenance process, simply because we just spend so much time in maintenance mode.

Some examples

Skipping the planning stage.

This is generally caused by the "Weeks of coding can save days of planning" fallacy.  Uncover all the main parts of your design before you code yourself into a corner that requires a near-rewrite to escape.

Unit-tests are your insurance that future changes don't cause non-obvious regressions.  They also have a magical way of creating better factored, higher-quality code.  If you skip this stage, your code will not be factored as well as it could be, and come maintenance time, you'll be scared to make changes for fear of breaking something somewhere else.

Writing classes and functions that do a lot

This one is a really seductive one.  We want to be consumers of classes and methods that do a lot for us, because higher levels of abstraction are how we get more done in less time.  From the consumer stand-point, this makes lots of sense, and it's actually the best way to improve development speed and maintenance speed at the same time.  If you're also the person writing the class or function that does more though, you have to be very careful to use the highest abstractions possible for that as well, even if you're writing its inner abstractions as well.  What this means is that every level of abstraction does as little as possible, and you build up to the level of abstraction that you want with those thin, thin layers.  Classes and functions that try to do a lot instead of relying on other classes and functions are very hard to maintain, simply because there's so much to do read and understand, and there's no easy way for the maintainer to skim and decide what piece is relevant, because the pieces are doing multiple things.

I'm talking about everything involving non-readable and poorly factored code, including bad names, foggy separation of concerns, etc.  Code should not be written just for the machine.  It needs to be written for the next guy that comes along as well.  That next guy could be you.  Or a violent psychopath

Using code generation

Code generation is just pure evil, if there's ever a remote chance a human will ever have to read it or change it.  A computer program doesn't know how to write code that a human will understand, and human understanding is exactly what we need to strive for if we care about making maintainable code.

Understatement of the day:  You're more than just a typist

It's largely irrelevant if you can touch type at 90 wpm.  It's largely irrelevant if your IDE can do automated refactors.  It's largely irrelevant if your efficiency with Emacs or vim makes you look like Neo in The Matrix.  These things are great bonuses, but won't make much difference to your proficiency (and efficiency) as a programmer if you neglect to first consider the maintainability of your codebase.  (Corollary:  In the name of all that is good in this world, please don't name your variables p1, atb, etc, or your functions like atoi() just for the sake of saving keystrokes!)

Time and time again my mistakes remind me:  there's only one way to be more productive over the long haul: work with better, higher abstractions.

Of course when debugging, you still need to get information about what's going on from somewhere; Usually for me this means relying heavily on print statements and logging.  I would say that the most interesting pieces of information to collect would be the values that methods are getting as input and giving as output, and what methods are called.  For example, for a function add() that adds two numbers, I do debugging log entries that look like this:

add called with *args: (1, 1), **kwargs: {}  --> 2

This is pretty simple stuff.  I'm sure lots of people write debug output like this.  The problem though is that it's a pain to sprinkle the print or log statements everywhere, and it's a pain to clean them up later.

Enter the Decorator

I got this idea from Bram Cohen's blog, but I have no idea if he actually does what I'm doing here: using Python's decorators feature to make logging of any method (add() in this example) as easy as slapping a "@log__calls_decorator" at the top of the method's definition, like this:

@log__calls_decorator
def add(operand1, operand2):
	return operand1 + operand2

So that's how the dream decorator would be applied.  This decorator annotation syntax is really just a shortcut for this:

add = log_calls_decorator(add)

You can see that add() is simply being replaced with whatever the output of log_calls_decorator(add) is.  The decorator must take a callable as input and provide one as output.  A decorator basically *replaces* the method it decorates, though *usually* in its replacement functionality, it also calls that method it is decorating.  That's how decorators "wrap" methods and it's why we pass the original add() to it: we want it to be able to still run add() at some point in its execution.

Let's just create a decorator that does nothing additional first:

def log_calls_decorator(f):
    def wrapped(*args, **kwargs):
        try:
            retval = f(*args, **kwargs)
			return retval
        except Exception, e:
            raise
    return wrapped

You can see that log_calls_decorator takes a function as input, calls it, and either returns its result or re-raises any exception that it raises.  It might seem unnecessary to catch an exception just to reraise it, but when we start to add additional functionality to the decorator, we'll want different output in the case of exceptions: Something like this:

add called with *args: (1, "blah"), **kwargs: {}  RAISED EXCEPTION TypeError, somefile.py, 20

All we need to do is add the extra functionality we want now (For the sake of simplicity, we'll just use print output instead of a logger, but changing this to use a logger should be elementary).

def log_calls_decorator(f):
    def wrapped(*args, **kwargs):
        call_string = "%s called with *args: %r, **kwargs: %r " % (f.__name__, args, kwargs)
        try:
            retval = f(*args, **kwargs)
            call_string += " --> " + repr(retval)
            print call_string
            return retval
        except Exception, e:
            top = traceback.extract_stack()[-1]   # get traceback info to print out later
            call_string += " RAISED EXCEPTION: "
            call_string += ", ".join([type(e).__name__, os.path.basename(top[0]), str(top[1])])
            print call_string
            raise
    return wrapped

Basically, with this little gem, I can just slap this decorator on any function I want to track as far as calls / inputs / outputs go!  Debugging is now one-step less frustrating and time-consuming.

A pattern is meant to be a common solution to a common problem.  Thinking in terms of patterns can be a good way to think (and talk) at a higher level of abstraction, so they can be good.  One major problem with patterns though is that they're always taken as gospel, as if they are The Correct Way, and they shouldn't be questionned.  Just because some pattern has a name and a standard implementation does not mean it's a "best practice".

Solutions Searching For Problems

It's a shame that patterns are named and cataloged only by what they are and do rather than by the problem that they're intended to solve.  This has a tendency to create a solution-in-search-of-a-problem mentality which can cause patterns to be implemented in places where they're not the best solution.  The problem needs to dictate the solution, not the other way around.

If you have a problem and you know of a pattern that solves it better than anything else you can conceive, then by all means use it.  Conversely, all patterns should be avoided unless they can be proven to be better than all other conceivable solutions.  Any pattern is the wrong solution if it's not the best solution to the actual problem.

In general, if you're not sure if a given pattern provides a solution that fits your problem, forget you ever even heard about the pattern and just write the solution from scratch.  You may find yourself eventually converging on a rewrite of the pattern, but more likely you'll come up with something better suited to your particular needs.  Most of the famous named patterns are really not all that revolutionary.

Three Cheers for Java

Secondly, patterns are often language-specific.  Because patterns are "solutions", and different languages have different capabilities and deficiencies, the patterns that are relevant from one language to the other are different.  Most of the famous named ones are from the GoF book and are thus Java-centric.

What that means is that they may or may not be useful in another language.  Some of them really are not.  For example, what use is the command pattern in a language with first class functions, closures, and the ability to pass functions as parameters?

Also, there are cases where a pattern is useful in your language of choice, but its prescribed implementation is different or entirely unnecessary.  For instance, python has decorators as a language feature, so some implementation of the Decorator pattern is simply obsolete.

Unquestionable Code

Thirdly, who's to say that the original author of the pattern isn't just wrong?  We're in real trouble if we start believing some programmer is infallible, even the ones we know to be better than us.

Patterns, like any other code, should be challenged as much as possible for your given problem in your given environment, and they should be used only when no better option is available.

Here's how it looks in Java:

public class Singleton {
  private static final Singleton INSTANCE = new Singleton();

  // Private constructor prevents instantiation from other classes
  private Singleton() {}

  public static Singleton getInstance() {
    return INSTANCE;
  }
}

This mechanism is definitely simple and effective at creating only one instance of itself, but it's quite deceptive in its simplicity. Many other languages don't have the ability to create private constructors, and that inability can reveal some general problems with the pattern. Here it is in Actionscript 3.0 (though I could've easily shown it in another language like PHP, Python, or Javascript):

public class Singleton {
  private static var instance:Singleton;
  private static var creatingSingleton:Boolean = false;

  public function Singleton(){
    if (!canConstruct) throw new Error( "Cannot instantiate" );
  }

  public static function getInstance():Singleton{
    if( !instance ){
      canConstruct = true;
      instance = new Singleton();
      canConstruct = false;
    }
    return instance;
  }
}

So if you have a DatabaseConnectionSingleton Here's how you'd use it:

var conn : DatabaseConnectionSingleton = DatabaseConnectionSingleton.getInstance();

With this algorithm, it becomes more obvious that we're trying to restrict construction. This code now needs to throw an exception if someone tries to instantiate it.

What's that smell?

It's an interesting exception to throw, because it's not really an "exceptional" or rare circumstance when this would get thrown -- it's a "never-ever-ever" or "something-is-wrong-in-the-universe" situation. There's no good reason to catch it and recover, because there's no reasonable way to reliably continue after this exception is thrown. And make no mistake, the java version's private constructor is highly unusual as well. I actually prefer this Actionscript version, because at least the weirdness is more obvious. Any highly unusual code like this should be considered a smell . It should force you to ask yourself why you don't you have to do things like this more often.  Indeed:

• Ordinarily you don't dictate how many times a class can be constructed.
• Ordinarily you don't restrict how a class is used when you write it.
• Ordinarily you don't actually add code to make a class less useful.

These are all things that should be considered very carefully. I would find it difficult to argue that you should make exceptions to your ordinary practices in this case. The poor guy who uses your singleton has to know how to instantiate it properly -- you left him with that responsibility so that he can avoid the Exception you throw that should never have occurred. Why not just leave him the responsibility of just instantiating the class once? In general: If you just want one instance of something, why not just create one instance of something? That seems so simple!

The Insidious Truth

The singleton pattern isn't really about ensuring that you only create one instance at all. If it was why didn't we just do this instead?

public class Singleton {
  private static var instantiated:Boolean = false;

  public function Singleton(){
    if (instantiated) throw new Error( "instance already exists" );
  }
}

// usage:    var conn : DatabaseConnectionSingleton = new DatabaseConnectionSingleton();

It's a simpler implementation, with a simpler usage, and it accomplishes the same instantiate-only-once criteria as the previous algorithm.

This is where the true nature of the singleton is revealed. This second implementation is missing a key part of the functionality that Singleton users are really after: the instance is statically bound to the class so that the instance is globally available.

Do I smell something else?

Having an object globally available is very strange indeed. Java doesn't have globals at all. PHP intentionally uses the globals keyword so that if you want to bring something into a method's scope that is not an explicit parameter or an instance variable, you can't do it without first declaring it. Global state is generally as shunned as much as goto itself, and for good reasons:

Non-locality. Global state is by definition not local. The reason you care is because having data far from where it is acted upon adds complexity. You have to keep track of what else might be modifying that data, and what those modifications might mean for your code. In a large system, global state increases complexity quite significantly as the maintainers have to track more and more places where global state can be modified and more and more places where those changes might have ramifications. The pursuit of locality is the primary reason for object-oriented programming in general.

Dependencies. If an object requires a singleton to perform it's duties, then it depends on that singleton. Anywhere you use your object, you must make sure you also use the singleton. That's increased coupling which decreases the ease with which you re-use your object. Your object always needs that singleton around any time you want to use it. That's painful when re-using your object in other applications, and it's painful when you want to write unit-tests. That singleton needs to travel everywhere your object goes.

In short, the singleton pattern makes code more complex, less useful, and a real pain to re-use or test. Eliminating singletons can be tricky, but it's a worthwhile endeavour.