Habit is habit, and not to be flung out of the window by any man, but coaxed down-stairs a step at a time.
– Mark Twain, “Pudd’nhead Wilson’s Calendar
In my last entry I talked about how difficult it is to transition a team from being a deadline-oriented team to a quality-oriented team. The summary: It’s extremely difficult to change to a quality-centric approach, simply because it’s a massive cultural change. It requires the formation of entirely new habits, and the casting-away of old ones. Many developers will resist this change in every way, because it’s just completely foreign.
I wrote about 12 stages of unit-testing consciousness as well:
- run existing tests
- maintain existing tests
- understand the importance of code coverage
- write new tests for code that is written by someone else who understands testability
- understand the problems of dependencies and what constitutes testability
- write code and tests for scenarios with no heavy dependencies
- understand the role of mocking / faking
- write code and tests for scenarios with heavy dependencies by mocking / faking
- understand the limitations of coverage analysis
- understand the value of writing tests before code
- refactor legacy code to be testable in ways that aren’t necessarily safe
- refactor legacy code to be testable in ways that are safe
In order to get a developer to that 12th stage of consciousness, that developer has to go through each stage, one at a time; Some of them can be pretty tricky to pass through too. Let’s talk about how to do that as gently as possible.
1. Running Existing Tests
In order to get to stage 1, the team needs to know how to run the existing tests. Of course, it’s essential to first have existing tests. This is where we get our first prerequisite: The Testing Guru.
The Testing Guru
The Testing Guru is someone who can preach the word of TDD, and knows how to answer the hard questions that will come along the way. This person will have to be comfortable setting up the initial framework and teaching the others on the team how to get through each and every one of these stages. Quite frankly, I think that if you don’t have a developer that is already knowledgeable in the area of TDD, you’re probably not going to succeed at getting anyone on the team to stage 1 or beyond. It’s just too tough of a switch without good solid guidance, and occasional hand-holding.
Assuming you’ve got a testing guru, and that person has started writing some tests, everything is so much easier if you’re committed to automated continuous integration (CI) and have a continuous integration server that can run the tests for you. This is a huge help because instead of trying to start a new policy/habit where people run all the tests, you can have the CI server do it for you, and have it report a build breakage whenever there’s a failure.
If you’re doing continuous automated testing, eventually people will start modifying code in ways that requires the tests to change. Before the team gets to that point, the guru should let them know how to run the tests themselves, and inform them not to commit any code if there’s a failing test (because that will be considered a broken build). Stage 1 has basically been reached when everyone is running the tests often enough to not break the build.
In my experience, if the tests are not run as part of the CI server’s work, they’ll simply be neglected over time, and the testing guru will repeatedly find broken tests that he’ll need to fix, because no one else will be bought-in to the process (yet).
2. Maintaining Existing Tests
At some point before the entire team is comfortable with running the tests, some tests will get broken and they’ll be blocking a developer from committing his work. When these come up, the guru should be ready to do some pair programming and show the developer around the test structure and how to fix the broken test. Enough sessions like this will lead to developers who can fix simple broken tests on their own.
This is a bit of a large step. As developers maintain tests, they learn to write tests. They’ll start to see what more complex tests look like. They’ll start to understand the fundamentals of testability. This stage could take some time for the entire team to achieve.
Information Radiator #1: Test Count
An information radiator is basically a succinct and impact-full visual representation of some information, preferably in a central public place where everyone can see it. It could be as simple as a graph that you print-out and pin to the wall. The information that we’d like to show is the trend of the change in the number of tests. At this point, the CI server should be reporting on the number of tests that are in your test suite. The team should commit to making sure that that number doesn’t regress. If you set up an information radiator to track the test count over time, when someone solves a broken test by taking the easy route and removing it instead of fixing it, it will be an obvious dip in the line chart. More visualizations like this will be necessary in the next stage too…
3. Understanding the importance of Code Coverage
This is another stage where the CI server plays a critical role: You’ll need to get code coverage reporting integrated with the rest of the build process.
Information Radiator #2: Code Coverage %
The reports should be as easily available as possible to everyone. This should probably be via a website (so developers can drill in to see branch-by-branch coverage metrics), as well as an information radiator. Most developers will genuinely be interested in this stat, so if you can make it obvious and public they’ll pay attention to it, and be more apt to want to contribute to improving it. It’s important too to track the change-trend of coverage over time, so developers can start to understand what factors effect it and how.
Once they see the holes in coverage in the current tests, they’ll be interested in plugging those holes. They’ll see that coverage itself is a good metric for measuring how well testing is going. Putting a simple metric to the concept like that can have a crystallizing effect.
Developers might want to run the coverage tools locally as well to help them find holes in their testing before they commit. The guru should be ready to help with that.
4. Writing new tests for code that is written by someone else who understands testability
As the team sees that code coverage number fluctuate, they’re incentivized to begin writing their own new tests. This is where our team introduced the one-unit-test-per-code-commit rule. It basically goes like this:
There must be at least one new unit-test in every code commit (No excuses).
There might be some developers on the team who have managed to avoid fixing existing tests while other developers have progressed onto this stage. This one-unit-test-per-code-commit rule will bring those developers out of the wood-work, so that they can be brought gently to this step.
There will be times that a code change is so insignificant that a new test doesn’t make sense in that area of the code base. That’s okay: in order to satisfy this rule, developers should be free to find the easiest piece of code anywhere in the codebase to write the most trivial testcase they can. It doesn’t matter at all if it’s not a high-value test! Keep that in mind. This is supposed to be a gentle introduction. The guru should be available to help developers find those dead-simple pieces of code (and possibly demo how the coverage tools can be useful for identifying them).
The entire team should be looking for the one test in each peer code review as well, so they hold each other responsible.
The guru should be ready to help developers that want to test some more difficult piece of code. He should pair program with any developers to help them advance as quickly as possible through the concepts associated with writing tests (There’s no reason to have the entire team advance at the pace of the lowest common denominator).
Over time, the test count trend will slope upward, and you might even see a positive trend in the code coverage percent (though don’t expect it!). The team will start to get more comfortable with writing tests and using the coverage tools (to find the easy ones). Eventually the code with easy-test potential will start to be harder and hard to find as those specimens all get under test. This will push developers to try writing tests for code that is increasingly difficult.
5. Understanding the problems of dependencies and what constitutes testability
As developers work through writing their tests for the 1-test rule, they’ll start to realize that dependencies and coupling in general make testing hard. There will be parts of the codebase that they won’t know how to test, and they’ll start to see the patterns in why that code is so hard to test.
This experience will lead them to an understanding of what constitutes testability.
6. Writing Code and Tests for Scenarios with No Heavy Dependencies
When developers reach stage 5, they’ll start to write their new code in ways that are more testable (ie sprout classes and methods, as defined in Working Effectively with Legacy Code). They’ll start to ask questions about how to write code that is more testable. They’ll start to ask how the more difficult concepts could possibly be tested.
With the understanding from stage 5, they’ll be ready to start writing code and tests for scenarios that don’t have heavy dependencies (like the database!). The guru should be available to help them through those concepts, probably through pair-programming.
Developers at this stage will start to feel a much larger degree of confidence with code that is more heavily tested, whether it’s their own code, or they’re maintaining code written by another developer. They’ll also start to see how much nicer their code is to maintain when it’s been written with testing in mind.
7. Understanding the role of mocking / faking
By the time developers are writing their own tests, they have probably already seen (and maybe maintained) some tests written by the guru that use mocks and fakes. They’ll see how mocks can swap out their more tricky dependencies so they can have fast tests that run without a lot of overhead, and so they concentrate on the code-under-test.
They’ll get comfortable with the mocking framework and be ready for the next step…
8. Writing code and tests for scenarios with heavy dependencies by mocking / faking
This step is pretty much self-explanatory, but it’s another big one. Developers that reach this stage will still be thinking of testing as the second stage in the process, but they’ll be able to struggle through writing testable code and the tests for it in new-code scenarios. They’ll also be able to factor some code out of the existing codebase and bring that code under test.
At this point the team should commit to ensuring that the coverage percent should not regress. It might even be useful to have the CI server report a broken build in the case of this regression. That way everyone has the same interests in understanding the concept of code coverage and the tools for evaluating it. The team certainly has all the skills at this point to ensure their own success.
The coverage percent trend should only go up at this point, but that will get more and more difficult, pushing the developers into subsequent stages.
9. Understanding the limitations of coverage analysis
Along the way, some developers might notice that they get bugs logged against code that they have 100% under test. This is normal of course, because 100% code coverage doesn’t guarantee correctness, it just guarantees that the code is being 100% exercised, and the code is 100% testable. While those are extremely useful metrics for determining how well a set of tests cover the behaviour that’s expected, developers should understand that there still might be holes in their tests. Any bugs that are found in 100% covered code will indicate the need for still another test. Fortunately if that portion of the code is 100% under-test, it will be easy to add a new test.
Information Radiator #3: bug count
Alright, you’re probably already keeping track of this, but it’s really useful at this stage to show the trending of the bug-count and how it corresponds to the trends in number of tests, and code coverage.
10. Understanding the value of writing tests before code
Many developers won’t ever get 100% into test-first development, but they’ll all realize that code that is written without testing in mind is often really hard to test. Tests must at least be considered before (and while) the code is being written. The guru might help illustrate this concept with the use of ping pong pair programming. It’s basically TDD in a pair programming scenario with a bit of a fun competitive twist.
11. Refactoring legacy code to be testable in ways that aren’t necessarily safe
Developers will naturally start to refactor existing code so that it’s testable. They’ll more-than-likely do this in a way that is not very safe at all given the chicken-egg scenario where you shouldn’t refactor code that isn’t already under test, and you can’t easily get some code under test until it’s refactored.
As the team digs further and further into the legacy code to bring it under test, they will start to see that they’re causing new bugs in the process. Ideally, the guru should try to help the team avoid this stage entirely, but I included it because it seems like every team goes through it, and struggles within it. The bug count metric might provide some impetus for getting into the next stage.
12. Refactoring legacy code to be testable in ways that are safe
The guru should show all the weird tricks (ie “seams” in Working Effectively with Legacy Code) to get some previously untestable code into a test harness somehow so that it can first be tested before it’s even modified.
The team should commit to not making changes without first having the code under test.
It’s not until this point that the team is experienced enough to be able to set code coverage goals for itself and commit to a schedule of coverage increases. The coverage trend will be invaluable in setting realistic goals.
In Closing, Some Caveats:
- In my experience, a team is unlikely to complete all 12 stages in a matter of weeks. It might be possible in a matter of months, but that would be pushing it.
- The quality of estimates for how long work will take will suffer as developers get used to how much time it takes at first, and later how much time it saves.
- Any return to deadline-driven development, no matter how brief, completely disrupts and undermines the process.