5 Common Excuses for Avoiding Unit-Testing
I just finished reading an article on the The Maintainability of Unit Tests (http://java.dzone.com/articles/maintainability-unit-tests) and a lot of what was being said there is contrary to my experiences with unit-testing, so I wanted to suggest a few counterpoints.
Basically there's a discussion of how unit-testing can hinder productivity when you have to make major changes to code under test, and more than a few commenters have questioned a goal of 100% coverage. I'd like to go through those points one-by-one, as well as through some additional points I hear quite often.
= External functional tests would allow more maintenance flexibility under the covers while still finding regressions. =
Here's a quote I love to drag around: “In the industry, people often go back and forth about whether particular tests are unit tests. Is a test really a unit test if it uses another production class? I go back to the two qualities: Does the test run fast? Can it help us localize errors quickly?” (Michael Feathers, Working Effectively with Legacy Code)
So, unit-tests are simply tests that run fast and help you quickly pinpoint where an error originates from. External functional tests are neither fast nor useful at telling you where the problem occurred. The end result is that you can't run your external functional test suite as you work in small iterations improving your codebase. You can't run it after every compile and before every check-in, because it takes too long! And when a functional test suite finds a bug, it can certainly tell you a bug exists, but it's completely useless and showing you where that bug occurred.
* Major changes to a heavily unit-tested portion of code can lead to tedious test changes that take up as much as 85% of the effort.
For new code: Instead of doing Test-first development, or test-driven design, our team often postpones writing tests until (1) the requirements have been completely fleshed-out and (2) the approach is known to be a good one. I think that's perfectly fine -- we're prototyping (http://caines.ca/blog/programming/the-lost-art-of-prototyping/). Ideally a prototype would be largely discarded when it's done and TDD would begin for the production-quality code (It would probably be pretty wasteful to not copy/paste large chunks from the prototype code though). It's critically important that you don't try to prototype your way to production-quality code, so it's best to stop prototyping as soon as the unclearness of the task becomes clear. After the point that the requirements and the approach are clear, any programming that you do without tests will just be the kind of cowboy-coding you were hoping to avoid.
For code already in production: Any major change should be executed as a series of minor changes. Every minor change should be preceded by introducing a failing test. If you follow those two rules, you predictably and systematically meet your goals while guaranteeing that you don't introduce regressions. It's worth it to take the time to write the kind of code that doesn't need to be revisited over and and over in the future, because it was done properly the first time while you had the best understanding of it. Count yourself lucky that you had all of these existing tests that probably just require minor tweaks. Take note of how little manual testing you had to do, and how few regressions you introduced.
* It's impossible and fruitless to unit-test view components / UI controls.
It's certainly not impossible. In our actionscript / flex code, we routinely get 100% coverage on UI controls. UI controls inevitably have logic and state, and it's not really something we try to avoid. A checkbox for instance has a state (checked or not) and it has presentation logic to decide whether to show the little black "X" or not. To move that logic and state somewhere else would be silly and to throw up our hands and say "We can't unit-test UI controls!" would be untrue. It's true that there are important (visual) functions of that control that still fall outside the realm of unit-testability, but that doesn't mean you can't get that code 100% covered by tests.
* Some code is so trivial that there's no point testing it.
One example given to illustrate this point was a DTO with 20+ properties on it. What's the point of testing that those properties can be written and read? It's so trivial that it can't be wrong!
Actually that's not true. For example: properties always have initial values, and it's easy for a piece of code to rely on a specific default initial value that could be changed by future maintenance. That's a breakage that static analysis (the compiler) isn't going to know about. Sure this kind of bug is unlikely to occur in any particular commit, but over time it becomes more and more likely, and your unit tests will always be there to prevent it.
It doesn't matter how trivial a piece of code is: humans can always find ways to get a bug into it.
It's also still very easy to get 100% coverage on that class without writing tests for it at all. It can be tested indirectly by getting all classes that use it 100% under test. If you can't get it 100% under test by getting its consumers under test, then it likely has unnecessary features or dead code. Clean that unnecessary maintenance overhead out and you'll get that 100%. Is it still a unit-test if you don't have that class directly covered? Don't worry about that -- ask yourself if that class is covered directly well enough for you to find defects in it quickly. If it's still not, then we're likely not talking about a trivial piece of code (like a DTO) at all.
* There's a law of diminishing returns as you write more and more minor tests to get 100% coverage.
In practice, we actually see the reverse. The devil does appear to be in the details: those things that we thought were minor understandably didn't get much thought during implementation either, and we often don't find bugs in our code (if we're doing post-implementation unit-testing) until we actually get down to those minor details. We're routinely surprised at the issues that we find in the last 20% of our coverage pursuit.
* Unit-testing is not a guarantee of correctness, so we shouldn't get too extreme about it.
I don't want to get too far into explaining the problems with this, but let me just propose the following:
"Ropes can sometimes break and snap, so mountain climbers shouldn't be too strict about using them."
It's absolutely true that unit-testing (even with 100% coverage) does not guarantee correctness. Anyone who's ever done any significant unit-testing has to wonder at some point "Who tests the test?". A unit-test is simply a double-check. If you have bad tests, passing them doesn't mean anything. Unit-testing is not a silver bullet that will solve all your quality problems: it's just an *extremely* valuable tool.
In my experience some of the nastiest issues lie in the parts that you think are too obvious to need testing. Unit-testing has been a huge boost to the level of quality of our output, so I see absolutely no reason to not aim for 100%.
I just finished reading an article on the The Maintainability of Unit Tests and a lot of what was being said there is contrary to my experiences with unit-testing, so I wanted to suggest a few counterpoints.
Basically there's a discussion of how unit-testing can hinder productivity when you have to make major changes to code under test, and more than a few commenters have questioned a goal of 100% coverage. I'd like to go through those points one-by-one, as well as through some additional points I hear quite often.
"External functional tests would allow more maintenance flexibility under the covers while still finding regressions."
Here's a quote I love to drag around:
“In the industry, people often go back and forth about whether particular tests are unit tests. Is a test really a unit test if it uses another production class? I go back to the two qualities: Does the test run fast? Can it help us localize errors quickly?” (Michael Feathers, Working Effectively with Legacy Code -- a really great book in my opinion!)
So, unit-tests are simply tests that run fast and help you quickly pinpoint where an error originates from.
External functional tests, on the other hand, are neither fast nor useful at telling you where the problem occurred. The end result is that you can't run your external functional test suite as you work in small iterations improving your codebase. You can't run it after every compile and before every check-in, because it takes too long! And when a functional test suite finds a bug, it can certainly tell you a bug exists, but it's completely useless and showing you where that bug occurred.
"Major changes to a heavily unit-tested portion of code can lead to tedious test changes that take up as much as 85% of the effort"
There are two different scenarios where I've seen this statement applied:
For new code
Instead of doing Test-first development, or test-driven design, our team often postpones writing tests until (1) the requirements have been completely fleshed-out and (2) the approach is known to be a good one. I think that's perfectly fine -- we're prototyping.
Ideally a prototype would be largely discarded when it's done and TDD would begin for the production-quality code (It would probably be pretty wasteful to not copy/paste large chunks from the prototype code though). It's critically important that you don't try to prototype your way to production-quality code, so it's best to stop prototyping as soon as the unclearness of the task becomes clear. After the point that the requirements and the approach are clear, any programming that you do without tests will just be the kind of cowboy-coding you were hoping to avoid.
For code already in production
Any major change should be executed as a series of minor changes. Every minor change should be preceded by introducing a failing test. If you follow those two rules, you predictably and systematically meet your goals while guaranteeing that you don't introduce regressions.
It's worth it to take the time to write the kind of code that doesn't need to be revisited over and over in the future, because it was done properly the first time while you had the best understanding of it.
Count yourself lucky that you had all of these existing tests that probably just require minor tweaks. Take note of how little manual testing you had to do, and how few regressions you introduced.
"It's impossible and fruitless to unit-test view components / UI controls."
It's certainly not impossible. In our actionscript / flex code, we routinely get 100% coverage on UI controls. UI controls inevitably have logic and state, and it's not really something we try to avoid. A checkbox for instance has a state (checked or not) and it has presentation logic to decide whether to show the little black "X" or not. To move that logic and state somewhere else would be silly and to throw up our hands and say "We can't unit-test UI controls!" would be untrue. It's true that there are important (visual) functions of that control that still fall outside the realm of unit-testability, but that doesn't mean you can't get that code 100% covered by tests.
"Some code is so trivial that there's no point testing it."
One example given to illustrate this point was a DTO with 20+ properties on it. What's the point of testing that those properties can be written and read? It's so trivial that it can't be wrong!
Actually that's not true. For example: properties always have initial values, and it's easy for a piece of code to rely on a specific default initial value that could be changed by future maintenance. That's a breakage that static analysis (the compiler) isn't going to know about. Sure this kind of bug is unlikely to occur in any particular commit, but over time it becomes more and more likely, and your unit tests will always be there to prevent it.
It doesn't matter how trivial a piece of code is: humans can always find ways to get a bug into it.
It's also still very easy to get 100% coverage on that class without writing tests for it at all. It can be tested indirectly by getting all classes that use it 100% under test. If you can't get it 100% under test by getting its consumers under test, then it likely has unnecessary features or dead code. Clean that unnecessary maintenance overhead out and you'll get that 100%. Is it still a unit-test if you don't have that class directly covered? Don't worry about that -- ask yourself if that class is covered directly well enough for you to find defects in it quickly. If it's still not, then we're likely not talking about a trivial piece of code (like a DTO) at all.
"There's a law of diminishing returns as you write more and more minor tests to get 100% coverage."
In practice, we actually see the reverse. The devil does appear to be in the details: those things that we thought were minor understandably didn't get much thought during implementation either. We often don't find bugs in our code (if we're doing post-implementation unit-testing of legacy code) until we actually get down to those minor details. We're routinely surprised at the issues that we find in the last 20% of our coverage pursuit.
"Unit-testing is not a guarantee of correctness, so we shouldn't get too extreme about it."
I don't want to get too far into explaining the problems with this, but let me just propose the following:
"Ropes can sometimes break and snap, so mountain climbers shouldn't be too strict about using them."
It's absolutely true that unit-testing (even with 100% coverage) does not guarantee correctness. Anyone who's ever done any significant unit-testing has to wonder at some point "Who tests the test?". A unit-test is simply a double-check. If you have bad tests, passing them doesn't mean anything. Unit-testing is not a silver bullet that will solve all your quality problems: it's just an extremely valuable tool.