The purpose of branching is to delay integration of some code into the main code line (for various reasons). There's a heavy cost that comes from this in that the less often you merge, the harder it is to merge. I'm blown away by the effectiveness of automatic conflict resolution in git, but the bottom line is that if two developers are working in the same area of the codebase, semantic conflicts will emerge that need the intervention of a human to resolve. These can be complicated and tedious and involve multiple developers. Yes, branching is easier with DVCS, but this actually exacerbates a fundamental merging problem that DVCS doesn't solve.

"What if I keep my branches short-lived?"

There are some better defenses for branching, like "What if I keep my branches very short-lived?", and while it does mitigate most of the problems with merging, it also mitigates any advantage you think you might get from branching. If you're only going to branch for a day, what's the point?

"Then how can I pick and choose what gets added/released?"

Merging early and often leads to simpler merges, but what about the case when you get partway through a feature and want to scrap it because you've come up with a better way? There are a number of ways to mitigate this problem, but if your need to "unmerge" occurs more often than your need to merge, you've probably got larger organizational issues to deal with. If those issues are purely technical, try more prototyping to uncover those issues earlier, and feature-toggles for better control over what gets released. If those organizational issues are non-technical, like frequent changing requirements mid-feature-development then you simply need to get better (and probably smaller) user stories before commencing.

"What if I do daily merges from the mainline into my feature branch?"

You'd think it shouldn't matter which way you merge if you merge often, but that's only fine if you're the only one branching. As soon as someone else on the team comes along and starts up their own feature branch and follows your same best practice, the two of you are diverging from each other and creating merge-debt.

Using a continuous integration server is NOT sufficient for actually practicing continuous integration.

I've heard of some teams using a bot to auto-add all branches to the team's CI server, making sure that each branch gets put through the proper rigor of the CI server. This is actually cargo-cult continuous integration though because those teams are not actually continually integrating at all. Merging early and often is a prerequisite of continuous integration, by definition. Instead, running multiple branches in CI supports maintaining those multiple distinct branches for even longer, so you're actually using the CI server to undermine the ideals of CI.  Even if you run a special build that tries to auto-merge all branches (yeah right!), but is not the mainline and is not intended for release, you lose the ability to regularly deliver work to some staging environment for manual regression testing.

A small non-distributed team doesn't have the same problems as Linus

If your team is in the same room and practices collective code ownership (instead of having a central maintainer) you should just do in-person peer code reviews (or better yet, pair programming) instead of relying on pull requests. It's wasteful to force every aspect of intra-team communication to go through the computers in the room. I would even go so far as to say that if you're in the same room, you don't actually need a DVCS.

Most of the reasons behind feature branches seem to be a cure that's worse than the disease itself. It's vastly simpler for us to commit to the mainline at all times, early and often, and deal with deactivating or removing particular pieces of code in other ways (like feature-toggles) as those problems arise (which does not in my experience tend to be very often compared to how often I am merging new stuff in that stays in).

I don't mean to be negative about DVCS either. I actually use git exclusively now, I think github has been a huge boon to open source, and I love anything that makes Linus more effective at maintaining my favourite OS kernel. It just really seems like feature-branches have never solved any real problem that I have, while they have caused a lot of pain.

Of course everyone wants to use JSON because of its simplicity, but that simplicity seems to be the cause of deficiencies in being truly RESTful.  I have to wonder how many other people are really trying to create RESTful JSON APIS when these deficiencies are largely unsolved, or solved only by ad-hoc means.

Deficiency #1: JSON is not Hypermedia

Nowhere in the JSON spec does it talk about how to represent relationships to other hypermedia documents with links.  Sure, we can add that, but we have to agree on how first, or we can't build generic clients.

Originally, I thought to go with a link format like this:

 { "self" : "/example", "next" : "/example?page2" }

Pros:

• The names in the name-value pairs define the relationship ("rel" in XML link nodes).
• It works nicely in a list.
• It's very json-esque in that it's untyped, like all other values in json.

Cons:

• It's very json-esque in that it's untyped.  This means that a client can't tell if it's a hyperlink or not without out-of-band knowledge.

That con leads me to believe that this format is unsuitable for hypermedia.

The second idea I happened across was this one :

{"link" : { "href" : "/example", "rel" : "self"}}

PROS:

• It clearly lets the client know that it's dealing with a link

CONS:

• You can't have more than one of these at the same level of a hash because names are supposed to be unique.  Soooo... You end up having to wrap these in an array and name that array.  Wow, that's going to be unnecessarily painful to parse.

The third idea I stumbled across and decided was right for me was the one described in the json-schema draft :

   {
     "links": [
       {
         "rel": "self"
         "href": "{id}"
       },
       {
         "rel": "up"
         "href": "{upId}"
       },
       {
         "rel": "children"
         "href": "?upId={id}"
       }
     ]
   }

PROs:

• Clearly indicates that we're dealing with hyperlinks
• Allows you to list links with the flattest possible structure
• Is in a draft for a real standard

CONs:

• Your links must always be in a array.

I can live with that con.

EDIT (Jan 8th 2012):  I came up with a new way that I think is even better, which better takes clients into consideration.  Here it is:

   {
     "links": {
       "self" : { "href": "{id}" },
       "up" : { "href": "{upId}" },
       "children" : { "href": "?upId={id}" }
     }
   }

This way makes it easy for the client to pull out URLs (eg  doc.links.self.href ) instead of having to search an array, and basically has all the other pros listed in other examples by sacrificing some flatness and using the relationship as the key instead of as part of the link hash.

Deficiency #2:  JSON lacks meta features

JSON hasn't got a standard way of specifying types (like links), or namespaces (for microformats, or embedding other JSON documents), and does anyone really use json schemas?  Those would be useful for specifying subsets of JSON that a client could be programmed to understand in advance... but there goes that simplicity of JSON.

Something makes me think that JSON will need to get vastly more complex if we're really going to use it as part of the semantic web and actually have intelligent automatons understand it.  The people that care about that are in the vast minority at the moment though, and I've actually got real work to do...

I've also seen that when anyone happens to cite a failed experience with Agile, the explanation from Agilists is invariably the cryptic "you weren't doing it right". This is a bit of an irritating response to anyone that's spent a significant amount of time following the ceremonies and speaking the lingo though.  Instead I'd like to submit that you have a much better chance of succeeding with Agile if you know the real reasons behind the madness.

There are a number of Agile values I've noticed in my years of practicing scrum and XP that I think stand on their own even if you're not doing Agile and just want to improve your development process in some small way.  I'm going to list them (and try to stay jargon-free) because I imagine that they would be useful individually, but they do have a sort of self-reinforcing quality to them where their effects will somehow be amplified if you use them together.  You should also feel free to break the rules where you feel they need to be broken, but I've very rarely found that to be necessary.  Usually when I've broken these rules, I've regretted it later, because what I thought was an exception didn't really end up being an exception.

Create a team, not just a group of individuals.

A project will move faster when it is being developed by a team that self-organizes around its goals rather than a group of individuals working separately.  Here are few rules of thumb to creating a team instead of just having a group of individuals:

• Don't assign tasks to individuals.  Let the team figure out who works on what, and let the team take responsibility for all tasks rather than individuals.   Everyone on the team needs to agree on what it means to be done and how, and everyone on the team needs to take responsibility for it.
• Strive for cross-functional skillsets.   Let the team members teach each other so they can better help each other in the future rather than allowing one person to become the bottleneck for any particular kind of task.
• Try not to change team size or membership very much.  A good team can be a fragile thing.
• Throw out the notion of individual code ownership. The entire team owns the code.
• Create more of a bullpen environment where the team members are all basically in the line of sight of one another and can just talk to one another about what they're working on.  This was tough, because it made us give up our individual offices, telecommuting, and deciding our own hours, but it was well worth it for the improvement to the quality of our work day.  Almost anytime a developer has to use some means of communication other than his voice to talk with a teammate, you have unnecessarily wasted time.
• Put QA personnel on the team so that testing can be can be on-going and so the team can more easily take on QA and testing as a team-wide responsibility.
• The whole team needs to agree on what kind of work is necessary for each task before it is to be considered "done".  (coding standards, documentation standards, how rigorously they test, etc, etc)

The result of this was a highly capable team that could "swarm" to complete goals reliably and quickly.  We actually developed a sort of sports team mentality that made this way of working genuinely fun, and helped us push ourselves to be better all the time.  There are a lot of challenges to creating a great team that can inter-communicate openly and honestly, but it's an amazing thing to behold when it happens.

Estimates need to come from developers.

This is can be very hard for some organizations to swallow, but developers are the ones that are most qualified to do estimations, and as professionals, they should be working diligently to improve their estimation capability.  When a developer is responsible for estimation, that developer feels a sense of ownership about his work and will feel a vastly greater sense of responsibility to meet the estimate.  Developers that have never learned how to estimate will not be good at it at first, but there's no other way for them to get good at it than to have them do it.  There are a few major things that can vastly help estimation too:

• Always break work down into tiny chunks.  We generally broke work up into chunks that were less than 3 days of effort.  I've never seen a task that couldn't be broken down into subtasks that are this small or smaller, if the developers are so inclined.
• The team should take a look at bad estimates in retrospect and figure out where they went wrong.
• Recognize work that will be impossible to do due to lack of clarity from the business, or unmet dependencies and push to have those requirements met.
• For highly speculative work, just commit to doing a proof-of-concept, or time-boxed research.
• The more developers that are involved in the estimation process, the better.  You'll probably want to include QA/testers as well, because good ones have unique insights.

Context-switching should be avoided mercilessly.

This means that each developer needs the discipline to make sure he only works on one task at a time, that the team works on as few tasks at one time, and that developers need sustained uninterruptable timeboxes where the priority of work is not allowed to change.

Most people (especially non-developers) have no idea how much of a huge time-suck that context switching is, and few realize how seldom it's actually worthwhile.  Also if the team is tackling as few tasks as possible at once, if any task can be finished by having more developers on it at once, then that's what the team should do rather than starting as many tasks as possible and working on them in parallel.  This might sound counter intuitive, but with the team ultra-focussed on as few tasks as possible at once, they can get the totality completed faster.

Try to always improve.

The team must commit to continually improving. At regular intervals, the team should look at how things are going and pick a few actionable things to make their work better/faster/easier.  You'd be surprised at the huge speed improvements that can come from this practice over time.  If the team has a shared calendar, set up a reoccurring meeting at a regular interval to have this meeting, and be laser-focussed on deciding what improvement you want to achieve and how exactly you'll try to achieve that.  We've always set aside 1 hour per week, but your mileage may vary.  Start every meeting by evaluating how successful the team was with the previous meeting's goals.

If something is difficult or time-consuming, do it more often.

This sounds crazy to the uninitiated but there are two reasons for this:

1. Doing difficult things more often will drive out new creativity in making them less difficult.  Pro-tip: Automation is almost always the answer when the problem is testing (See TDD, BDD, ATDD) or deployment.  (See Continuous Delivery)
2. Many types of tasks get more difficult the longer they are postponed and so are always easier if you do them more often instead of letting the risk of complexity pile up.  Integrating code into the main branch is one example  (See Continuous Integration).  Deploying is another (See Continuous Delivery).

Be merciless about achieving high quality

Over the long haul, speed comes from quality, so everyone on the team needs to make quality their job.  Everyone needs to test, and everyone needs to think about how to continually make testing easier without cutting corners.

• Have QA personnel on every team, and have them testing as the team works.
• The whole team needs to agree on what level of quality is needed, and what quality metrics will be used to judge every check-in.
• Perform a root-cause-analysis on each defect.  Try to figure out how to avoid that class of defect in the future.

Do everything in tiny increments.

Tiny increments reduce the risk of compounding problems, and allow you to get feedback and correct your course more often.

• break-down your work into tiny tasks
• plan just-in-time and just enough to do work.
• test as you go, instead of in huge batches
• merge code to the mainline early and often.
• deploy early and often.
• don't add anything to the system that you don't need until you need it (YAGNI)

Get customer feedback early and often

Get customer feedback early and often to converge on the best possible solution as fast as possible by constantly correcting course instead of blindly following some long-running plan or doing unnecessary work.  The customer's priorities and vision of the product will change as he sees results, so it's better to not find out about these changes after a lot of unnecessary work has been done.

• Do work in priority order as dictated by the customer.  This will allow them to see the aspects that they care about most as early as possible, and therefore help you converge on a unified vision of the product faster.
• Work break-down is better done in vertical slices where the customer can get some amount of real value from each piece (or at least see some evidence of work) rather than having many non-user-facing tasks, where possible.  This allows for more potential for feedback.
• The work shouldn't be considered done by anyone on the team until the customer, or a representative of the customer agrees that it's done.
• Constantly improve estimation skills so that they are generally more reliable so the customer knows the cost of the work he has requested and can make realistic decisions based on that.

Anyway, feel free to try what you want and throw out the rest!

Deadlines have a lot of real plusses:

• The business knows to expect *something* at that time that resembles the original request.  This makes the development process seem somewhat more reliable.
• The programmer gets a sense of clarity about his task when "doneness" can be defined with just a date, and some resemblance to the sliding requirements.
• The programmer gets a sense of urgency, and therefore feels motivated.
• The programmer doesn't have to bother doing the hard work of learning to estimate properly, and doesn't have to feel personally responsible for poor estimates.

I don't mean that with any kind of sarcasm either.  It may sound machiavellian to say things like "deadlines motivate programmers", but the fact of the matter is that they do (they're not the only way however -- I'll get to that later).  In fact, the programmer himself can be the biggest proponent of deadlines, if he has only worked in deadline-driven environments.  That type of programmer will in fact cling to deadlines, because for the aforementioned reasons (people really do want to feel motivated at work).  Without them, he ends up drifting through the day, and not knowing when to stop working on a given task.  He certainly has never learned to estimate properly, and is all-too-happy to never have to bother trying to push-back against management on unrealistic deadlines.

So there are real reasons why deadline-driven development is attractive to all people involved, and it should be no surprise that many software development shops just go from one deadline to the next in a fashion that can only be described as deadline-driven development.  There are certain realities that exist for these shops though, regardless of whether they're recognized:

• Most employees don't want to work beyond their agreed-upon 8 hrs very often.  A system that tries to run counter to this will have high employee turn-over, and therefore incur heavy employee on-boarding costs on a regular basis.
• Technical debt will be incurred constantly.  This is a perilous practice that makes your development process slower over time until the team can no longer develop seemingly basic features in any sort of reasonable time.  It's dangerous because it's very hard to even notice that it is happening until it's too late.

These results create a feedback loop that only serves to amplify the problem:  Employee turnover and on-boarding costs both increase as technical debt increases.  Developers that are new to the codebase tend to create more technical debt as well.  Because of this, the deadline-driven approach to software development can quickly spiral into dead-end codebases that are so mired in technical debt and complexity that no one wants to work on them, and they can't be given new features easily.

Obviously I always go on and on about quality and how it's the limiting factor in being as fast as possible, but there are some necessary trade-offs sometimes when you're developing software in the business world.

For example, when I used to do freelance contract work, I would get a lot of contracts to create one-off microsites for companies that wanted to run online promotional contests.  These microsites would need to go live on a not-too-far-away date where they would run for a few weeks before being closed down and deleted.  I was still concerned about quality, because word-of-mouth is how I got contracts, but technical debt wasn't any concern at all; I knew the software would just be deleted after a month, and I could do manually tested copy/paste spaghetti code to my heart's content if it meant I'd be done faster. *

Also: If I was running a start-up, for instance, I would probably incur more technical debt than if I was working on software that was profitable and had hundreds of users.  It makes more sense in the start-up scenario, because in that scenario, everything is a prototype -- you're not sure if the idea itself is even worthwhile.  You don't need to fight technical debt to ensure a fast future, because you need to be fast NOW, and you might not even have a future.  (Don't get me wrong though… I'd still do a heck of a lot of TDD  and aim for high coverage so I don't have to repeatedly test things manually, but then again I've got a lot of experience with automated testing, and I can write tests quickly enough now that it's an obvious time-saving over manual testing.)

So there are certainly some times to willingly take on technical debt.  There's a great blog post by Steve McConnell on this.

The cold hard facts of the matter though are that if a start-up ends up being successful while taking on technical debt, that technical debt will cost them over time, regardless of whether they realize it or not, or whether they decide to do something about it or not.   It should be obvious that on a long-lived codebase, a team is faster over-all when it minimizes technical debt at every step, rather than letting quality be the dimension that slides as a deadline approaches.  Deadline-driven development is basically a way of borrowing against the future speed of development to meet a certain date. It is tempting because it almost always works over the short-term, but over the long term all the codebases I've seen that are continually developed this way grind the team to a virtual halt at some point.

The corollary to this is that over the long term, a team should theoretically be able to develop faster without deadlines. That will almost certainly seem like insanity to anyone that's only done deadline-driven development, but in my experience it's true; I've worked on such a team, and while it was extremely fast, it was also an easily sustainable pace where everyone was happy with the results.

What does it take to get there?

• Everyone needs to realize that deadlines almost always incur technical debt and that technical debt is a way of borrowing against future development speed.
• Developers need to learn a new definition of done so that their work has a clear point of completion and so that it includes making sure technical debt is not accidentally incurred.
• Developers need to learn how to estimate reliably against this new definition of done.
• Developers need to find a new source of urgency to stay motivated.  (In reality though, meeting their own estimates is usually challenge enough!)
• Everyone needs to acknowledge that estimates come from the developers.
• Everyone needs to figure out how to keep management up to date on the real progress.
• For the most part, management needs to be content with monitoring the speed of progress rather than trying to dictate it.

I'm not trying to say these are easily attainable goals, but I do know that once you've attained them, the team moves very fast.  I basically liken deadline-driven development to a rolling wheel that often stops or goes in reverse briefly compared to a wheel in a more quality-driven approach that may move forward more slowly, but never stops or rolls backward.  In my experience the slower moving wheel that never rolls back has a faster net speed and is more reliable and manageable for the business.

* There's a flip-side to that story too though… After 4 or 5 of those microsites I started to see a pattern in the requirements and started to build up some higher-quality libraries I could use to implement a new site easily to the extent that a site that would take me 3 days would later take me 3 hours.).  You can be sure that I spent a lot of time making that library super DRY and well (unit) tested because that was code that had a future.

I see a lot of people finding their automated tests to be tedious and time-consuming to write and wondering what parts are okay to not test.   A QA manager once asked me a question along the same lines… "How much coverage do you really need?"  I basically answered "What percent do you not care about verifying?"

There are arguments that go the other way:

"Some code is so trivial, that it need not be tested."

Sorry -- I just don't buy it.  One character off in a source file can bring an entire app to its knees.  Any developer that's been coding for any length of time knows this.

"Some code is so low value, that it need not be tested."

Then why are you writing it?  Seriously… if it doesn't matter if it works or not, just delete it.  There's no company that wants you to spend time writing code that no one finds any value in.

"There are diminishing returns as you approach complete coverage, and the effort to do so increases".

I don't buy this one either.  Many people buy into testing the happy paths through the code, but feel they can save time by not testing the error scenarios.  The truth of the matter though is that people are generally pretty lazy about error scenarios in their application code, and there are tonnes of bugs that lie in there as a result.  I usually find the happy path scenarios are well thought out and very few bugs occur there, but the really tricky bits lie in how the code behaves under exceptional and error scenarios.  That can get complicated, and getting it wrong is almost always the same as not handling errors at all.

If you decide that you want to write tests for everything (instead of manually testing everything before each check in), you still have the problem that testing this extremely can be tedious and time-consuming.

The answer is not to test less, but to test smarter:

1. Don't test the same thing twice. If you've tested something to the best of your ability, consider that behaviour tested and don't retest those details in some other kind of test (like an integration test that uses it).  For example, in Rails it's common to find people testing that the built-in validators on their model work, when those validators have been tested to death by the rails core team.  Just test that the proper validators *exist* in your model, and consider that part of the model testing done.
2. Be completely intolerant of boilerplate application code. If you find yourself writing the same stuff over and over, even if it's not pure copy/paste, figure out a way to put that in a library, or a super class or some other form of abstraction.  Then test the hell out of that abstraction, so that every time you use it, you're using something that you don't need to test again.  If you can see repetitive patterns in your application code, even if it's not pure cut and paste, you will have unnecessary repetitive patterns in your test code as well.  Keeping your application code DRY is a great way to make sure your test code is DRY as well.
3. Be completely intolerant of repetitive test code. Learn about how to build more powerful abstractions in your tests as well.  You can...
• • build better matchers (like hamcrest in java or shoulda in ruby)
  • build more powerful assertions, and other helper methods that can be reused
  • use a framework with more configurable 'contexts' than just the usual n-unit 'setup' function
  • use more powerful libraries for setting up test data (like Ruby's FactoryGirl)
  • use the most powerful mocking/stubbing framework you can find for your platform

Test code needs to be DRY too, or it will get painful very quickly.

4. Mock only when you need to. Some people have a tendency to try to mock everything a class uses, but that takes a lot of time, and it makes the tests more fragile.  You also introduce the possibility that the mock doesn't properly mimic the dependency it's supposed to, practically guaranteeing false-positive tests.  You have two other choices:

After having been a freelance developer for about 9 years, I wouldn't recommend it to anyone, but there were a few things I learned in that time that I try to take to my subsequent jobs.

• Learn what "done" means. When you say something's done, it means no more time needs to be spent on it in order for a customer to use it.  People will expect that when you say "done" you're using the same definition as the one in the English dictionary, and not just the works-on-my-machine imaginary definition.
• Be able to justify your technological decisions. The business's primary purpose is making money, NOT making software so even technological decisions ultimately need to have financial justifications.  "Because it's cool" can't be used to justify any technology decision, and generally makes people stop listening to you.
• Make your work process visible, reliable, and honest. The business would almost always prefer that you spend the necessary time to make your process visible and reliable (reliable estimation, task breakdown, etc), than complete sooner.  Don't be that developer that tries to avoid all reporting for a week in the name of saving time, because everyone knows you really just don't like reporting.
• Put effort into all areas of responsibility, not just programming. There are many concerns that a developer must be responsible for other than programming (requirements, specs, testing, deployment, maintenance).  Yes, testing is a large part of the developer's job.  Yes, deployment is a large part of the developer's job.
• Get over the fact that you have to plan and go to meetings. Working with a company requires more rigor, process and communication overhead than working by yourself (meetings, planning, etc).  You should always strive for the lightest possible process that works, but there is always process, even when some would like to pretend that there isn't.  The general rule is that as the number of people increases, so will the number of meetings and the amount of planning that are necessary to coordinate them.
• Don't whine about someone else's code. Everyone already knows that you think that that code that someone else wrote is garbage, because nearly every developer says that about nearly every piece of code he/she did not write.  If it's a real impediment, be specific and constructive.  Offer real solutions other than jumping straight to "rewrite".   A call for a greenfield rewrite is almost always the knee-jerk reaction of the lazy-minded.
• Don't say that something can't be done when it can just to influence engineering or product direction (even when it's really hard). You lose credibility for future consultation quickly that way.
• Constantly improve your estimation skills. Spend the time to break your work down into small enough chunks that they can be reliably estimated.  In the process of working on those chunks, if you learn that your estimate is slipping, report up early and often.  Look back on your failed estimates and figure out what went wrong.  People will only believe your "I can clone google in a weekend" estimates for so long before they decide you don't know how to estimate.
• Don't go to your manager/client with problems unless you're bringing solutions as well (and preferably more than one). You were hired to be the problem solver.
• Keep an eye on your results because over the long haul, results are the only thing that matter. Even if every task you're on fails for some reason that you explain is out of your control, you'll likely be dismissed.  If the reasons really are out of your control, you should really leave first anyway.

One fine day in the trenches, a junior engineer was tasked with optimizing a particular web service call that was way too slow (sometimes slower than 20 seconds!).  I had just figured out how to use a performance profiler for that particular platform, so I thought I'd offer to show him how to use it.  His response was that he was on a tight deadline and didn't have time to learn to use the profiler, and that he was just going to tune the database calls to make them perform better.

Later in the year when the problem still wasn't solved, I had a chance to take it on myself.  The profiler showed me that this service call was itself making around 20 web service calls to a second system that could obviously add up to 20 seconds, if that other web service took 1 second for each of its responses (and the logs showed that sometimes it did).  It turned out that almost all of those web service calls were redundant and completely unnecessary.  Simply removing them (which took about 1 day of effort) shaved 19 seconds off this web service call's duration in production in some cases.

The total execution time spent on database calls was 5 milliseconds.  A 50% improvement there would probably take a week's worth of effort, and really only get you 2 or 3 milliseconds of performance improvement.

The moral of the story is that there's a pretty simple process for optimizing a system:

Skipping step 1 isn't a shortcut.  It's absolutely critical to know that you're optimizing the bottleneck, and not some other random component.

This is pretty obvious stuff for most experienced developers of course.  And while they seem to know to apply this to the systems that they work on, it's more rare that they notice it outside of the code and see it in the process of software engineering in general.

The software development process is a system as well, made up of components.  Requirements come in at one end, and deliverables go out the other end.  As people who are involved in that process, we naturally want it to be more and more efficient.  We are constantly trying to figure out how to optimize that system so we can deliver better software in less time.

So step 1:  Identify and isolate the component that is tying up the most time in the system.

All of these time consumers are affected directly or indirectly by our ability to control quality.  In my experience, the majority of the software development process isn't hung up in requirements analyis, design, or implementation.  It's hung up in all these Quality Control related issues.  Quality Control is the constraint that keeps us from creating the right software faster.  Steve McConnell confirms it in a more scientific way than I ever could:

The rest of us would do well to learn from a discovery made by IBM in the 1970s: Products with the lowest defect counts also have the shortest schedules (Jones 1991). Many organizations currently develop software with defect levels that give them longer schedules than necessary. After surveying about 4000 software projects, Capers Jones reported that poor quality was one of the most common reasons for schedule overruns (1994). He also reported that poor quality is implicated in close to half of all canceled projects. A Software Engineering Institute survey found that more than 60 percent of organizations assessed suffered from inadequate quality assurance (Kitson and Masters 1993). On the curve in Figure 1, those organizations are to the left of the 95-percent-removal line.

(http://www.stevemcconnell.com/articles/art04.htm -- The whole article is extremely illuminating)

What this means is that if we really want to move quickly, we need to take a good hard look at how we manage quality.  Of course this is counter-intuitive, and its actually hard to swallow, because ensuring quality can be mundane and tedious if you're not doing it right.  But as developers we'd be silly to ignore the real bottleneck in the process, and optimizing for that.  Micro-optimizations elsewhere in the process will yield very little benefit compared to improving measures that ensure quality.

As developers, we love to argue about and discuss these micro-optimizations all day long:

It sure seems silly though to be pursuing micro-optimizations, at the expense of tackling the real #1 time-suck:  Quality.

It's no fun to write about stuff that everyone agrees on, so I'm going to write about something that almost everyone seems to disagree with me about:

100% code coverage is almost always possible. It's a valuable target, and it's a worthwhile goal in and of itself.

The primary argument that gets dragged out repeatedly is that 100% code coverage doesn't indicate anything about the quality of your tests whatsoever. It's certainly possible to get code 100% covered without writing a single assert, or by writing incorrect or incomplete tests. If you don't actually assert everything you care about properly, code coverage doesn't seem to be very valuable at all.

While there is a lot of truth to this, unless you have dishonest developers, that code coverage percent does indicate an attempt at quality tests. And really, an honest attempt is all we can ever ask for.

I'm willing to hypothesize though, for the sake of argument that somehow the class under test somehow is 100% covered by wrong tests, incomplete tests, or tests with no asserts whatsoever.  Let's call it an Exercise Suite instead, because that's all it does; it exercises the code.

I'm going to still contend that that coverage metric is a quality indicator. The important thing to notice is that it means you've somehow managed to exercise 100% of your codebase.  This is actually a huge accomplishment that means a few things:

(1) You don't have dead code in your class-under-test.

If you can't exercise every branch in a given class, you've got dead code. Dead code should always be removed to keep from cluttering up future maintenance efforts. (Think you might need it again? That's what version control systems are for -- You can always go back and get it again.)

(2) Your codebase can be loaded into a test harness.

This is actually one of the hardest parts about testing codebases written in procedural languages, or languages that allow scripts, or "includes".  The contents of a given file can't be brought under test reliably and repeatably unless it is possible to load it into the test harness without it actually doing anything until it's exercised by the tests. If you've ever tried to get a legacy PHP project under test, you'll know what I'm talking about here.

(3) You've figured out how to avoid/mitigate side effects.

Code that accesses a database, the filesystem, a web service, hardware, etc is very hard to exercise quickly and without side effects. If your exercise suite has managed to get 100% coverage, while still running fast, reliably and repeatably, you've scored a major victory.

(4) You can detect a myriad of run-time errors.

"Failures" are generally what you have when a test fails. You care about "errors" too though: Errors are all the things that can go wrong at run-time that are obvious just by executing the code with certain inputs. Exercise alone can uncover a number of these. We routinely catch null pointer exceptions in our test suite (attempts to access properties that aren't correctly initialized yet), not through failed tests, but instead simply through exercise. The test runner catches many of them, and always shoves those directly in our face to be solved.

I think my unpopular opinion comes from two things about me that puts me in a bit of a minority as far as developers go:

(1) I've spent a bunch of time actually doing TDD, and I expect unit-tests to look like they were written before the code even if they weren't. If you're writing tests before code, as TDD prescribes, you're generally not going to end up with uncovered code. At any point in the process, the only asymmetry that's likely is that there are tests that are trying to test code that doesn't yet exist. To have uncovered code in TDD generally means you've done something wrong.

(2) I'm extremely accustomed to working with dynamically-typed languages, so I've learned to rely on unit-tests for almost every aspect of verification. I use them to debug. I use them for syntax-validation. I use them to ensure I'm interacting with other objects in ways that they expect (ie, "Did I get that parameter order right in that method call?"). There is no static type-system, so I don't have any quality safe-guard other than the tests I write. In that scenario, uncovered code is a risk I don't want to take -- it could even contain syntax errors!

In short, the hypothetical Exercise Suite has a lot of value in and of itself.

I just finished reading an article on the The Maintainability of Unit Tests and a lot of what was being said there is contrary to my experiences with unit-testing, so I wanted to suggest a few counterpoints.

Basically there's a discussion of how unit-testing can hinder productivity when you have to make major changes to code under test, and more than a few commenters have questioned a goal of 100% coverage.  I'd like to go through those points one-by-one, as well as through some additional points I hear quite often.

"External functional tests would allow more maintenance flexibility under the covers while still finding regressions."

Here's a quote I love to drag around:

“In the industry, people often go back and forth about whether particular tests are unit tests. Is a test really a unit test if it uses another production class? I go back to the two qualities: Does the test run fast? Can it help us localize errors quickly?” (Michael Feathers,  Working Effectively with Legacy Code -- a really great book in my opinion!)

So, unit-tests are simply tests that run fast and help you quickly pinpoint where an error originates from.

External functional tests, on the other hand, are neither fast nor useful at telling you where the problem occurred.  The end result is that you can't run your external functional test suite as you work in small iterations improving your codebase.  You can't run it after every compile and before every check-in, because it takes too long!  And when a functional test suite finds a bug, it can certainly tell you a bug exists, but it's completely useless and showing you where that bug occurred.

"Major changes to a heavily unit-tested portion of code can lead to tedious test changes that take up as much as 85% of the effort"

There are two different scenarios where I've seen this statement applied:

For new code

Instead of doing Test-first development, or test-driven design, our team often postpones writing tests until (1) the requirements have been completely fleshed-out and (2) the approach is known to be a good one.  I think that's perfectly fine -- we're prototyping.

Ideally a prototype would be largely discarded when it's done and TDD would begin for the production-quality code (It would probably be pretty wasteful to not copy/paste large chunks from the prototype code though).  It's critically important that you don't try to prototype your way to production-quality code, so it's best to stop prototyping as soon as the unclearness of the task becomes clear.  After the point that the requirements and the approach are clear, any programming that you do without tests will just be the kind of cowboy-coding you were hoping to avoid.

For code already in production

Any major change should be executed as a series of minor changes.  Every minor change should be preceded by introducing a failing test.  If you follow those two rules, you predictably and systematically meet your goals while guaranteeing that you don't introduce regressions.

It's worth it to take the time to write the kind of code that doesn't need to be revisited over and over in the future, because it was done properly the first time while you had the best understanding of it.

Count yourself lucky that you had all of these existing tests that probably just require minor tweaks.  Take note of how little manual testing you had to do, and how few regressions you introduced.

"It's impossible and fruitless to unit-test view components / UI controls."

It's certainly not impossible.  In our actionscript / flex code, we routinely get 100% coverage on UI controls.  UI controls inevitably have logic and state, and it's not really something we try to avoid.  A checkbox for instance has a state (checked or not) and it has presentation logic to decide whether to show the little black "X" or not.  To move that logic and state somewhere else would be silly and to throw up our hands and say "We can't unit-test UI controls!" would be untrue.  It's true that there are important (visual) functions of that control that still fall outside the realm of unit-testability, but that doesn't mean you can't get that code 100% covered by tests.

"Some code is so trivial that there's no point testing it."

One example given to illustrate this point was a DTO with 20+ properties on it.  What's the point of testing that those properties can be written and read?  It's so trivial that it can't be wrong!

Actually that's not true.  For example: properties always have initial values, and it's easy for a piece of code to rely on a specific default initial value that could be changed by future maintenance.  That's a breakage that static analysis (the compiler) isn't going to know about.  Sure this kind of bug is unlikely to occur in any particular commit, but over time it becomes more and more likely, and your unit tests will always be there to prevent it.

It doesn't matter how trivial a piece of code is: humans can always find ways to get a bug into it.

It's also still very easy to get 100% coverage on that class without writing tests for it at all.  It can be tested indirectly by getting all classes that use it 100% under test.  If you can't get it 100% under test by getting its consumers under test, then it likely has unnecessary features or dead code.  Clean that unnecessary maintenance overhead out and you'll get that 100%.  Is it still a unit-test if you don't have that class directly covered?  Don't worry about that -- ask yourself if that class is covered directly well enough for you to find defects in it quickly.  If it's still not, then we're likely not talking about a trivial piece of code (like a DTO) at all.

"There's a law of diminishing returns as you write more and more minor tests to get 100% coverage."

In practice, we actually see the reverse.  The devil does appear to be in the details:  those things that we thought were minor understandably didn't get much thought during implementation either.  We often don't find bugs in our code (if we're doing post-implementation unit-testing of legacy code) until we actually get down to those minor details.  We're routinely surprised at the issues that we find in the last 20% of our coverage pursuit.

"Unit-testing is not a guarantee of correctness, so we shouldn't get too extreme about it."

I don't want to get too far into explaining the problems with this, but let me just propose the following:

"Ropes can sometimes break and snap, so mountain climbers shouldn't be too strict about using them."

It's absolutely true that unit-testing (even with 100% coverage) does not guarantee correctness.  Anyone who's ever done any significant unit-testing has to wonder at some point "Who tests the test?".  A unit-test is simply a double-check.  If you have bad tests, passing them doesn't mean anything.  Unit-testing is not a silver bullet that will solve all your quality problems: it's just an extremely valuable tool.

Habit is habit, and not to be flung out of the window by any man, but coaxed down-stairs a step at a time.
-- Mark Twain, "Pudd'nhead Wilson's Calendar

In my last entry I talked about how difficult it is to transition a team from being a deadline-oriented team to a quality-oriented team.  The summary: It's extremely difficult to change to a quality-centric approach, simply because it's a massive cultural change.  It requires the formation of entirely new habits, and the casting-away of old ones.  Many developers will resist this change in every way, because it's just completely foreign.

I wrote about 12 stages of unit-testing consciousness as well:

1. run existing tests
2. maintain existing tests
3. understand the importance of code coverage
4. write new tests for code that is written by someone else who understands testability
5. understand the problems of dependencies and what constitutes testability
6. write code and tests for scenarios with no heavy dependencies
7. understand the role of mocking / faking
8. write code and tests for scenarios with heavy dependencies by mocking / faking
9. understand the limitations of coverage analysis
10. understand the value of writing tests before code
11. refactor legacy code to be testable in ways that aren't necessarily safe
12. refactor legacy code to be testable in ways that are safe

In order to get a developer to that 12th stage of consciousness, that developer has to go through each stage, one at a time;  Some of them can be pretty tricky to pass through too.  Let's talk about how to do that as gently as possible.

1. Running Existing Tests

In order to get to stage 1, the team needs to know how to run the existing tests.  Of course, it's essential to first have existing tests.  This is where we get our first prerequisite: The Testing Guru.

The Testing Guru

The Testing Guru is someone who can preach the word of TDD, and knows how to answer the hard questions that will come along the way.  This person will have to be comfortable setting up the initial framework and teaching the others on the team how to get through each and every one of these stages.  Quite frankly, I think that if you don't have a developer that is already knowledgeable in the area of TDD, you're probably not going to succeed at getting anyone on the team to stage 1 or beyond.  It's just too tough of a switch without good solid guidance, and occasional hand-holding.

Continuous Testing

Assuming you've got a testing guru, and that person has started writing some tests, everything is so much easier if you're committed to automated continuous integration (CI) and have a continuous integration server that can run the tests for you.  This is a huge help because instead of trying to start a new policy/habit where people run all the tests, you can have the CI server do it for you, and have it report a build breakage whenever there's a failure.

If you're doing continuous automated testing, eventually people will start modifying code in ways that requires the tests to change.  Before the team gets to that point, the guru should let them know how to run the tests themselves, and inform them not to commit any code if there's a failing test (because that will be considered a broken build).  Stage 1 has basically been reached when everyone is running the tests often enough to not break the build.

In my experience, if the tests are not run as part of the CI server's work, they'll simply be neglected over time, and the testing guru will repeatedly find broken tests that he'll need to fix, because no one else will be bought-in to the process (yet).

2. Maintaining Existing Tests

At some point before the entire team is comfortable with running the tests, some tests will get broken and they'll be blocking a developer from committing his work.  When these come up, the guru should be ready to do some pair programming and show the developer around the test structure and how to fix the broken test.  Enough sessions like this will lead to developers who can fix simple broken tests on their own.

This is a bit of a large step.  As developers maintain tests, they learn to write tests.  They'll start to see what more complex tests look like.  They'll start to understand the fundamentals of testability.  This stage could take some time for the entire team to achieve.

Information Radiator #1:  Test Count

An information radiator is basically a succinct and impact-full visual representation of some information, preferably in a central public place where everyone can see it.  It could be as simple as a graph that you print-out and pin to the wall.  The information that we'd like to show is the trend of the change in the number of tests.  At this point, the CI server should be reporting on the number of tests that are in your test suite.  The team should commit to making sure that that number doesn't regress.  If you set up an information radiator to track the test count over time, when someone solves a broken test by taking the easy route and removing it instead of fixing it, it will be an obvious dip in the line chart.  More visualizations like this will be necessary in the next stage too...

3. Understanding the importance of Code Coverage

This is another stage where the CI server plays a critical role: You'll need to get code coverage reporting integrated with the rest of the build process.

Information Radiator #2: Code Coverage %

The reports should be as easily available as possible to everyone.  This should probably be via a website (so developers can drill in to see branch-by-branch coverage metrics), as well as an information radiator.  Most developers will genuinely be interested in this stat, so if you can make it obvious and public they'll pay attention to it, and be more apt to want to contribute to improving it.  It's important too to track the change-trend of coverage over time, so developers can start to understand what factors effect it and how.

Once they see the holes in coverage in the current tests, they'll be interested in plugging those holes.  They'll see that coverage itself is a good metric for measuring how well testing is going.  Putting a simple metric to the concept like that can have a crystallizing effect.

Developers might want to run the coverage tools locally as well to help them find holes in their testing before they commit.  The guru should be ready to help with that.

4. Writing new tests for code that is written by someone else who understands testability

As the team sees that code coverage number fluctuate, they're incentivized to begin writing their own new tests.  This is where our team introduced the one-unit-test-per-code-commit rule.  It basically goes like this:

There must be at least one new unit-test in every code commit (No excuses).

There might be some developers on the team who have managed to avoid fixing existing tests while other developers have progressed onto this stage.  This one-unit-test-per-code-commit rule will bring those developers out of the wood-work, so that they can be brought gently to this step.

There will be times that a code change is so insignificant that a new test doesn't make sense in that area of the code base.  That's okay: in order to satisfy this rule, developers should be free to find the easiest piece of code anywhere in the codebase to write the most trivial testcase they can.  It doesn't matter at all if it's not a high-value test!  Keep that in mind.  This is supposed to be a gentle introduction.  The guru should be available to help developers find those dead-simple pieces of code (and possibly demo how the coverage tools can be useful for identifying them).

The entire team should be looking for the one test in each peer code review as well, so they hold each other responsible.

The guru should be ready to help developers that want to test some more difficult piece of code.  He should pair program with any developers to help them advance as quickly as possible through the concepts associated with writing tests (There's no reason to have the entire team advance at the pace of the lowest common denominator).

Over time, the test count trend will slope upward, and you might even see a positive trend in the code coverage percent (though don't expect it!).  The team will start to get more comfortable with writing tests and using the coverage tools (to find the easy ones).  Eventually the code with easy-test potential will start to be harder and hard to find as those specimens all get under test.  This will push developers to try writing tests for code that is increasingly difficult.

5. Understanding the problems of dependencies and what constitutes testability

As developers work through writing their tests for the 1-test rule, they'll start to realize that dependencies and coupling in general make testing hard.  There will be parts of the codebase that they won't know how to test, and they'll start to see the patterns in why that code is so hard to test.

This experience will lead them to an understanding of what constitutes testability.

6. Writing Code and Tests for Scenarios with No Heavy Dependencies

When developers reach stage 5, they'll start to write their new code in ways that are more testable (ie sprout classes and methods, as defined in Working Effectively with Legacy Code).  They'll start to ask questions about how to write code that is more testable.  They'll start to ask how the more difficult concepts could possibly be tested.

With the understanding from stage 5, they'll be ready to start writing code and tests for scenarios that don't have heavy dependencies (like the database!).  The guru should be available to help them through those concepts, probably through pair-programming.

Developers at this stage will start to feel a much larger degree of confidence with code that is more heavily tested, whether it's their own code, or they're maintaining code written by another developer.  They'll also start to see how much nicer their code is to maintain when it's been written with testing in mind.

7. Understanding the role of mocking / faking

By the time developers are writing their own tests, they have probably already seen (and maybe maintained) some tests written by the guru that use mocks and fakes.  They'll see how mocks can swap out their more tricky dependencies so they can have fast tests that run without a lot of overhead, and so they concentrate on the code-under-test.

They'll get comfortable with the mocking framework and be ready for the next step...

8. Writing code and tests for scenarios with heavy dependencies by mocking / faking

This step is pretty much self-explanatory, but it's another big one.  Developers that reach this stage will still be thinking of testing as the second stage in the process, but they'll be able to struggle through writing testable code and the tests for it in new-code scenarios.  They'll also be able to factor some code out of the existing codebase and bring that code under test.

At this point the team should commit to ensuring that the coverage percent should not regress.  It might even be useful to have the CI server report a broken build in the case of this regression.  That way everyone has the same interests in understanding the concept of code coverage and the tools for evaluating it.  The team certainly has all the skills at this point to ensure their own success.

The coverage percent trend should only go up at this point, but that will get more and more difficult, pushing the developers into subsequent stages.

9. Understanding the limitations of coverage analysis

Along the way, some developers might notice that they get bugs logged against code that they have 100% under test.  This is normal of course, because 100% code coverage doesn't guarantee correctness, it just guarantees that the code is being 100% exercised, and the code is 100% testable.  While those are extremely useful metrics for determining how well a set of tests cover the behaviour that's expected, developers should understand that there still might be holes in their tests.  Any bugs that are found in 100% covered code will indicate the need for still another test.  Fortunately if that portion of the code is 100% under-test, it will be easy to add a new test.

Information Radiator #3: bug count

Alright, you're probably already keeping track of this, but it's really useful at this stage to show the trending of the bug-count and how it corresponds to the trends in number of tests, and code coverage.

10. Understanding the value of writing tests before code

Many developers won't ever get 100% into test-first development, but they'll all realize that code that is written without testing in mind is often really hard to test.  Tests must at least be considered before (and while) the code is being written.  The guru might help illustrate this concept with the use of ping pong pair programming.  It's basically TDD in a pair programming scenario with a bit of a fun competitive twist.

11. Refactoring legacy code to be testable in ways that aren't necessarily safe

Developers will naturally start to refactor existing code so that it's testable.  They'll more-than-likely do this in a way that is not very safe at all given the chicken-egg scenario where you shouldn't refactor code that isn't already under test, and you can't easily get some code under test until it's refactored.

As the team digs further and further into the legacy code to bring it under test, they will start to see that they're causing new bugs in the process.  Ideally, the guru should try to help the team avoid this stage entirely, but I included it because it seems like every team goes through it, and struggles within it.  The bug count metric might provide some impetus for getting into the next stage.

12. Refactoring legacy code to be testable in ways that are safe

The guru should show all the weird tricks (ie "seams" in Working Effectively with Legacy Code) to get some previously untestable code into a test harness somehow so that it can first be tested before it's even modified.

The team should commit to not making changes without first having the code under test.

It's not until this point that the team is experienced enough to be able to set code coverage goals for itself and commit to a schedule of coverage increases.  The coverage trend will be invaluable in setting realistic goals.

In Closing, Some Caveats:

• In my experience, a team is unlikely to complete all 12 stages in a matter of weeks.  It might be possible in a matter of months, but that would be pushing it.
• The quality of estimates for how long work will take will suffer as developers get used to how much time it takes at first, and later how much time it saves.
• Any return to deadline-driven development, no matter how brief, completely disrupts and undermines the process.