Deciding What Not to Test
I see a lot of people finding their automated tests to be tedious and time-consuming to write and wondering what parts are okay to not test.   A QA manager once asked me a question along the same lines… “How much coverage do you really need?”  I basically answered “What percent do you not care about verifying?”
There are arguments that go the other way:
“Some code is so trivial, that it need not be tested.”  Sorry — I just don’t buy it.  One character off in a source file can bring an entire app to its knees.  Any developer that’s been coding for any length of time knows this.
“Some code is so low value, that it need not be tested.”  Then why are you writing it?  Seriously… if it doesn’t matter if it works or not, just delete it.  There’s no company that wants you to spend time writing code that no one finds any value in.
“There are diminishing returns as you approach complete coverage, and the effort to do so increases”.  I don’t buy this one either.  Many people buy into testing the happy paths through the code, but feel they can save time by not testing the error scenarios.  The truth of the matter though is that people are generally pretty lazy about error scenarios in their code, and there are tonnes of bugs that lie in there as a result.  I usually find the happy path scenarios are well thought out
and very few bugs occur there, but the really tricky bits lie in how the code behaves under exceptional and error scenarios.  That can get complicated, and getting it wrong is almost always the same as not handling errors at all.
If you decide that you want to write tests for everything (instead of manually testing everything before each check in), you still have the problem that testing this extremely can be tedious and time-consuming.  The answer is not to test less, but to test smarter.
(1) Don’t test the same thing twice.  If you’ve tested something to the best of your ability, consider that behaviour tested and don’t retest those details in some other kind of test (like an integration test that uses it).  For example, in Rails it’s common to find people testing that the built-in validators on their model work, when those validators have been tested to death by the rails core team.  Just test that the proper validators *exist* in your model, and consider that model testing
done.
(2) Be completely intolerant of boilerplate application code.  If you find yourself writing the same stuff over and over, even if it’s not pure copy/paste, figure out a way to put that in a library, or a super class or some other form of abstraction.  Then test the hell out of that abstraction, so that every time you use it, you’re using something that you don’t need to test again.  If you can see repetitive patterns in your application code, even if it’s not pure cut and paste, you will have unnecessary repetitive patterns in your test code as well.  Keeping your application code DRY is a great way to make sure your
test code is DRY as well.
(3) Be completely intolerant of repetitive test code.  Learn about how to build more powerful abstractions in your tests as well.  You can…
- build better matchers (like hamcrest in java or shoulda in ruby)
- build more powerful assertions, and other helper methods that can be reused
- use a framework with more configurable ‘contexts’ than just the usual n-unit ‘setup’ function
- use more powerful libraries for setting up test data (like Ruby’s FactoryGirl)
- use the most powerful mocking/stubbing framework you can find for your platform
Test code needs to be DRY too, or it will get painful very quickly.
(4) Mock only when you need to.  Some people have a tendency to try to mock everything a class uses, but that takes a lot of time, and it makes the tests more fragile.  You also introduce the possibility that the mock doesn’t properly mimic the dependency it’s supposed to, practically guaranteeing false-positive tests.  You have two other choices:
- stubs.  Just as good as a mock for faking out a tough dependency, but it usually takes less setup and it won’t whine about how many times you call it.  Because of that, tests that use stubs are generally less brittle than tests that use mocks when application code changes happen.
- no fake at all.  There’s huge value in testing your units together, and as long as it’s still very fast and you can isolate defects quickly when a test fails, you have the option of not faking anything at all.  You’re often going to want to fake network connections, file i/o, and database access, but there usually isn’t much need for faking anything else.

I see a lot of people finding their automated tests to be tedious and time-consuming to write and wondering what parts are okay to not test.   A QA manager once asked me a question along the same lines… “How much coverage do you really need?”  I basically answered “What percent do you not care about verifying?”

There are arguments that go the other way:

“Some code is so trivial, that it need not be tested.”

Sorry — I just don’t buy it.  One character off in a source file can bring an entire app to its knees.  Any developer that’s been coding for any length of time knows this.

“Some code is so low value, that it need not be tested.”

Then why are you writing it?  Seriously… if it doesn’t matter if it works or not, just delete it.  There’s no company that wants you to spend time writing code that no one finds any value in.

“There are diminishing returns as you approach complete coverage, and the effort to do so increases”.

I don’t buy this one either.  Many people buy into testing the happy paths through the code, but feel they can save time by not testing the error scenarios.  The truth of the matter though is that people are generally pretty lazy about error scenarios in their application code, and there are tonnes of bugs that lie in there as a result.  I usually find the happy path scenarios are well thought out and very few bugs occur there, but the really tricky bits lie in how the code behaves under exceptional and error scenarios.  That can get complicated, and getting it wrong is almost always the same as not handling errors at all.

If you decide that you want to write tests for everything (instead of manually testing everything before each check in), you still have the problem that testing this extremely can be tedious and time-consuming.

The answer is not to test less, but to test smarter:

  1. Don’t test the same thing twice. If you’ve tested something to the best of your ability, consider that behaviour tested and don’t retest those details in some other kind of test (like an integration test that uses it).  For example, in Rails it’s common to find people testing that the built-in validators on their model work, when those validators have been tested to death by the rails core team.  Just test that the proper validators *exist* in your model, and consider that part of the model testing done.
  2. Be completely intolerant of boilerplate application code. If you find yourself writing the same stuff over and over, even if it’s not pure copy/paste, figure out a way to put that in a library, or a super class or some other form of abstraction.  Then test the hell out of that abstraction, so that every time you use it, you’re using something that you don’t need to test again.  If you can see repetitive patterns in your application code, even if it’s not pure cut and paste, you will have unnecessary repetitive patterns in your test code as well.  Keeping your application code DRY is a great way to make sure your test code is DRY as well.
  3. Be completely intolerant of repetitive test code. Learn about how to build more powerful abstractions in your tests as well.  You can…
      • build better matchers (like hamcrest in java or shoulda in ruby)
      • build more powerful assertions, and other helper methods that can be reused
      • use a framework with more configurable ‘contexts’ than just the usual n-unit ‘setup’ function
      • use more powerful libraries for setting up test data (like Ruby’s FactoryGirl)
      • use the most powerful mocking/stubbing framework you can find for your platform

    Test code needs to be DRY too, or it will get painful very quickly.

  4. Mock only when you need to. Some people have a tendency to try to mock everything a class uses, but that takes a lot of time, and it makes the tests more fragile.  You also introduce the possibility that the mock doesn’t properly mimic the dependency it’s supposed to, practically guaranteeing false-positive tests.  You have two other choices:
    • stubs. A stub is just as good as a mock for faking out a tough dependency, but it usually takes less setup and it won’t whine about how many times you call it.  Because of that, tests that use stubs are generally less brittle than tests that use mocks when application code changes happen.
    • no fake at all. There’s huge value in testing your units together, and as long as it’s still very fast and you can isolate defects quickly when a test fails, you have the option of not faking anything at all.  You’re often going to want to fake network connections, file i/o, and database access, but there usually isn’t much need for faking anything else.

Leave a reply

required

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>