There are a few major reasons for this:

The server is no place for view logic

The devices available for running a web app are vastly different from when these web frameworks first sprung up.  The slide towards thicker / richer clients has been proceeding on pace with increases in processing power since the Web 2.0 days.  It's much simpler to handle views and view logic in only one place, and that place is slowly moving away from the server side.  MVC has always been a strained pattern for a server-side, non-gui application and it is been a confusing and complicated trade-off to have the backend generating front-end logic.    Front-end frameworks like backbone.js, as well as advances in web technologies like HTML5's history.pushState are now making server-free views a realistic quality of cutting-edge front-ends.   Rendering on the client-side also gives us the opportunity to create responsive designs based on device capability rather than having the server try to somehow figure out what the capabilities are without actually running on that device.

The kinks aren't all the way out yet, but I do think the trend is clear.

Server-side Templating and .to_json are both underpowered and overpowered for the actual requirements of JSON APIs

There's no need for templating on the serverside (or view helpers, or any view-related cruft) to generate simple JSON documents, but there are a tonne of problems left unsolved when we fail to see that generating a JSON API is more than just a serialization problem.

How should dates look?  (RFC3339 / ISO8601 of course!).  What should the JSON error document look like when you provide a 400 and want to tell the client why?  How should links to other resources in the API look?  How does a collection look?  How does pagination look?

These aren't just serialization concerns, and they have nothing to do with templating.

HATEOAS is not just an academic pursuit

A thick client does not want to maintain a vast list of static strings representing all the crazy URLs that it will have to call in a non standard API.  As an API designer, you don't want them doing this anyway, because hard-coded URLs and URL structures make it a real pain for you to change your API.

The AtomPub protocol, if you ignore its XMLiness,  gets this right -- The thick client just knows the root URL (which serves up a simple service document) and is aware of a bunch of relationship types (the 'rel' attribute on 'link' and 'a' tags) that it can follow as necessary.  If a game client needs to access a list of players, it just goes to the root url, and follows the link with the rel property called 'players' (and probably remembers that link until the server says it has moved via 301 status code).  JSON has no concept of links or rel, but this is still easy to imagine and implement, and while it's a teeny bit of extra work up front, the standardization buys you the ability to start writing smarter HTTP/REST clients for your frontend that take care of much more for you automatically, so you can spend time on real business logic and actually do something more productive than fiddle with the javascript version of a routes.rb file.

(A really great API framework might generate some or all of these links on its own and automatically put them in the JSON documents.  It's pretty easy to imagine pagination links like next/back being generated automatically, or even links amoung resources in some cases, possibly based on some internally declared relationship.)

Rails-style MVC frameworks have a horrible routing solution for RESTFul JSON APIs

In a resource-oriented API, the router need not be concerned with the http methods that are or are not allowed for the resource.  That's the concern of the resource and the resource alone.  When the router tries to manage that, you get the unnecessary verbosity of a route for every method supported by the resource, and you get the app incorrectly throwing 404s instead of 405s when a method is not supported.  This probably means that 'controllers' need to go away in favor of 'resources', and routes can be vastly simplified, if not completely derived/inferred by the framework.  Because we keep thinking in this conventional MVC style though, we miss the possibility and potential of vastly more simple applications that actually do a lot more more for us.

The Application Developer shouldn't have to Deal with these Details

There's no  reason for us to all separately think about these problems and solve them in a million different ways every time we're confronted with them.  Aside from the years of wasted time this involves, we've also got a bunch of non-standard and sub-standard APIs to interact with, so all the client code needs to be custom as well and nothing is reusable.  This is why being RESTful is not just academic and this is why being concerned with the details is not pedantic.

As I said earlier, AtomPub gets a lot of this right.  A lighter-weight JSON equivalent would be a huge improvement to what people are doing today, because the conventions would mean that the framework can take care of most of these API details that we reimplement ad nauseum -- or worse, not at all.  It also means that frontends can start to abstract away more of the HTTP details as well.  This is already starting to happen in the new frontend MVC frameworks but in almost every case, the HTTP end of things still needs to be handled in a custom way for every API endpoint.  There is still way too much work left to the application developer, and we're silly to continue to do it over and over without coming up with a better abstraction.

CouchApps and WebMachine are just starting to touch on this style of architecture.  Backend-as-a-service platforms like Parse certainly understand how far this simple architecture can go, but ultimately there's a huge need for a framework that can create more complex RESTful APIs (in a language that's more general purpose and "blue collar" than Erlang).

Rails-style MVC frameworks are both too much, and not enough at the same time.  It really is time for a new framework to support this new architecture.

The purpose of branching is to delay integration of some code into the main code line (for various reasons). There's a heavy cost that comes from this in that the less often you merge, the harder it is to merge. I'm blown away by the effectiveness of automatic conflict resolution in git, but the bottom line is that if two developers are working in the same area of the codebase, semantic conflicts will emerge that need the intervention of a human to resolve. These can be complicated and tedious and involve multiple developers. Yes, branching is easier with DVCS, but this actually exacerbates a fundamental merging problem that DVCS doesn't solve.

"What if I keep my branches short-lived?"

There are some better defenses for branching, like "What if I keep my branches very short-lived?", and while it does mitigate most of the problems with merging, it also mitigates any advantage you think you might get from branching. If you're only going to branch for a day, what's the point?

"Then how can I pick and choose what gets added/released?"

Merging early and often leads to simpler merges, but what about the case when you get partway through a feature and want to scrap it because you've come up with a better way? There are a number of ways to mitigate this problem, but if your need to "unmerge" occurs more often than your need to merge, you've probably got larger organizational issues to deal with. If those issues are purely technical, try more prototyping to uncover those issues earlier, and feature-toggles for better control over what gets released. If those organizational issues are non-technical, like frequent changing requirements mid-feature-development then you simply need to get better (and probably smaller) user stories before commencing.

"What if I do daily merges from the mainline into my feature branch?"

You'd think it shouldn't matter which way you merge if you merge often, but that's only fine if you're the only one branching. As soon as someone else on the team comes along and starts up their own feature branch and follows your same best practice, the two of you are diverging from each other and creating merge-debt.

Using a continuous integration server is NOT sufficient for actually practicing continuous integration.

I've heard of some teams using a bot to auto-add all branches to the team's CI server, making sure that each branch gets put through the proper rigor of the CI server. This is actually cargo-cult continuous integration though because those teams are not actually continually integrating at all. Merging early and often is a prerequisite of continuous integration, by definition. Instead, running multiple branches in CI supports maintaining those multiple distinct branches for even longer, so you're actually using the CI server to undermine the ideals of CI.  Even if you run a special build that tries to auto-merge all branches (yeah right!), but is not the mainline and is not intended for release, you lose the ability to regularly deliver work to some staging environment for manual regression testing.

A small non-distributed team doesn't have the same problems as Linus

If your team is in the same room and practices collective code ownership (instead of having a central maintainer) you should just do in-person peer code reviews (or better yet, pair programming) instead of relying on pull requests. It's wasteful to force every aspect of intra-team communication to go through the computers in the room. I would even go so far as to say that if you're in the same room, you don't actually need a DVCS.

Most of the reasons behind feature branches seem to be a cure that's worse than the disease itself. It's vastly simpler for us to commit to the mainline at all times, early and often, and deal with deactivating or removing particular pieces of code in other ways (like feature-toggles) as those problems arise (which does not in my experience tend to be very often compared to how often I am merging new stuff in that stays in).

I don't mean to be negative about DVCS either. I actually use git exclusively now, I think github has been a huge boon to open source, and I love anything that makes Linus more effective at maintaining my favourite OS kernel. It just really seems like feature-branches have never solved any real problem that I have, while they have caused a lot of pain.

Of course everyone wants to use JSON because of its simplicity, but that simplicity seems to be the cause of deficiencies in being truly RESTful.  I have to wonder how many other people are really trying to create RESTful JSON APIS when these deficiencies are largely unsolved, or solved only by ad-hoc means.

Deficiency #1: JSON is not Hypermedia

Nowhere in the JSON spec does it talk about how to represent relationships to other hypermedia documents with links.  Sure, we can add that, but we have to agree on how first, or we can't build generic clients.

Originally, I thought to go with a link format like this:

 { "self" : "/example", "next" : "/example?page2" }

Pros:

• The names in the name-value pairs define the relationship ("rel" in XML link nodes).
• It works nicely in a list.
• It's very json-esque in that it's untyped, like all other values in json.

Cons:

• It's very json-esque in that it's untyped.  This means that a client can't tell if it's a hyperlink or not without out-of-band knowledge.

That con leads me to believe that this format is unsuitable for hypermedia.

The second idea I happened across was this one :

{"link" : { "href" : "/example", "rel" : "self"}}

PROS:

• It clearly lets the client know that it's dealing with a link

CONS:

• You can't have more than one of these at the same level of a hash because names are supposed to be unique.  Soooo... You end up having to wrap these in an array and name that array.  Wow, that's going to be unnecessarily painful to parse.

The third idea I stumbled across and decided was right for me was the one described in the json-schema draft :

   {
     "links": [
       {
         "rel": "self"
         "href": "{id}"
       },
       {
         "rel": "up"
         "href": "{upId}"
       },
       {
         "rel": "children"
         "href": "?upId={id}"
       }
     ]
   }

PROs:

• Clearly indicates that we're dealing with hyperlinks
• Allows you to list links with the flattest possible structure
• Is in a draft for a real standard

CONs:

• Your links must always be in a array.

I can live with that con.

EDIT (Jan 8th 2012):  I came up with a new way that I think is even better, which better takes clients into consideration.  Here it is:

   {
     "links": {
       "self" : { "href": "{id}" },
       "up" : { "href": "{upId}" },
       "children" : { "href": "?upId={id}" }
     }
   }

This way makes it easy for the client to pull out URLs (eg  doc.links.self.href ) instead of having to search an array, and basically has all the other pros listed in other examples by sacrificing some flatness and using the relationship as the key instead of as part of the link hash.

Deficiency #2:  JSON lacks meta features

JSON hasn't got a standard way of specifying types (like links), or namespaces (for microformats, or embedding other JSON documents), and does anyone really use json schemas?  Those would be useful for specifying subsets of JSON that a client could be programmed to understand in advance... but there goes that simplicity of JSON.

Something makes me think that JSON will need to get vastly more complex if we're really going to use it as part of the semantic web and actually have intelligent automatons understand it.  The people that care about that are in the vast minority at the moment though, and I've actually got real work to do...

I've also seen that when anyone happens to cite a failed experience with Agile, the explanation from Agilists is invariably the cryptic "you weren't doing it right". This is a bit of an irritating response to anyone that's spent a significant amount of time following the ceremonies and speaking the lingo though.  Instead I'd like to submit that you have a much better chance of succeeding with Agile if you know the real reasons behind the madness.

There are a number of Agile values I've noticed in my years of practicing scrum and XP that I think stand on their own even if you're not doing Agile and just want to improve your development process in some small way.  I'm going to list them (and try to stay jargon-free) because I imagine that they would be useful individually, but they do have a sort of self-reinforcing quality to them where their effects will somehow be amplified if you use them together.  You should also feel free to break the rules where you feel they need to be broken, but I've very rarely found that to be necessary.  Usually when I've broken these rules, I've regretted it later, because what I thought was an exception didn't really end up being an exception.

Create a team, not just a group of individuals.

A project will move faster when it is being developed by a team that self-organizes around its goals rather than a group of individuals working separately.  Here are few rules of thumb to creating a team instead of just having a group of individuals:

• Don't assign tasks to individuals.  Let the team figure out who works on what, and let the team take responsibility for all tasks rather than individuals.   Everyone on the team needs to agree on what it means to be done and how, and everyone on the team needs to take responsibility for it.
• Strive for cross-functional skillsets.   Let the team members teach each other so they can better help each other in the future rather than allowing one person to become the bottleneck for any particular kind of task.
• Try not to change team size or membership very much.  A good team can be a fragile thing.
• Throw out the notion of individual code ownership. The entire team owns the code.
• Create more of a bullpen environment where the team members are all basically in the line of sight of one another and can just talk to one another about what they're working on.  This was tough, because it made us give up our individual offices, telecommuting, and deciding our own hours, but it was well worth it for the improvement to the quality of our work day.  Almost anytime a developer has to use some means of communication other than his voice to talk with a teammate, you have unnecessarily wasted time.
• Put QA personnel on the team so that testing can be can be on-going and so the team can more easily take on QA and testing as a team-wide responsibility.
• The whole team needs to agree on what kind of work is necessary for each task before it is to be considered "done".  (coding standards, documentation standards, how rigorously they test, etc, etc)

The result of this was a highly capable team that could "swarm" to complete goals reliably and quickly.  We actually developed a sort of sports team mentality that made this way of working genuinely fun, and helped us push ourselves to be better all the time.  There are a lot of challenges to creating a great team that can inter-communicate openly and honestly, but it's an amazing thing to behold when it happens.

Estimates need to come from developers.

This is can be very hard for some organizations to swallow, but developers are the ones that are most qualified to do estimations, and as professionals, they should be working diligently to improve their estimation capability.  When a developer is responsible for estimation, that developer feels a sense of ownership about his work and will feel a vastly greater sense of responsibility to meet the estimate.  Developers that have never learned how to estimate will not be good at it at first, but there's no other way for them to get good at it than to have them do it.  There are a few major things that can vastly help estimation too:

• Always break work down into tiny chunks.  We generally broke work up into chunks that were less than 3 days of effort.  I've never seen a task that couldn't be broken down into subtasks that are this small or smaller, if the developers are so inclined.
• The team should take a look at bad estimates in retrospect and figure out where they went wrong.
• Recognize work that will be impossible to do due to lack of clarity from the business, or unmet dependencies and push to have those requirements met.
• For highly speculative work, just commit to doing a proof-of-concept, or time-boxed research.
• The more developers that are involved in the estimation process, the better.  You'll probably want to include QA/testers as well, because good ones have unique insights.

Context-switching should be avoided mercilessly.

This means that each developer needs the discipline to make sure he only works on one task at a time, that the team works on as few tasks at one time, and that developers need sustained uninterruptable timeboxes where the priority of work is not allowed to change.

Most people (especially non-developers) have no idea how much of a huge time-suck that context switching is, and few realize how seldom it's actually worthwhile.  Also if the team is tackling as few tasks as possible at once, if any task can be finished by having more developers on it at once, then that's what the team should do rather than starting as many tasks as possible and working on them in parallel.  This might sound counter intuitive, but with the team ultra-focussed on as few tasks as possible at once, they can get the totality completed faster.

Try to always improve.

The team must commit to continually improving. At regular intervals, the team should look at how things are going and pick a few actionable things to make their work better/faster/easier.  You'd be surprised at the huge speed improvements that can come from this practice over time.  If the team has a shared calendar, set up a reoccurring meeting at a regular interval to have this meeting, and be laser-focussed on deciding what improvement you want to achieve and how exactly you'll try to achieve that.  We've always set aside 1 hour per week, but your mileage may vary.  Start every meeting by evaluating how successful the team was with the previous meeting's goals.

If something is difficult or time-consuming, do it more often.

This sounds crazy to the uninitiated but there are two reasons for this:

1. Doing difficult things more often will drive out new creativity in making them less difficult.  Pro-tip: Automation is almost always the answer when the problem is testing (See TDD, BDD, ATDD) or deployment.  (See Continuous Delivery)
2. Many types of tasks get more difficult the longer they are postponed and so are always easier if you do them more often instead of letting the risk of complexity pile up.  Integrating code into the main branch is one example  (See Continuous Integration).  Deploying is another (See Continuous Delivery).

Be merciless about achieving high quality

Over the long haul, speed comes from quality, so everyone on the team needs to make quality their job.  Everyone needs to test, and everyone needs to think about how to continually make testing easier without cutting corners.

• Have QA personnel on every team, and have them testing as the team works.
• The whole team needs to agree on what level of quality is needed, and what quality metrics will be used to judge every check-in.
• Perform a root-cause-analysis on each defect.  Try to figure out how to avoid that class of defect in the future.

Do everything in tiny increments.

Tiny increments reduce the risk of compounding problems, and allow you to get feedback and correct your course more often.

• break-down your work into tiny tasks
• plan just-in-time and just enough to do work.
• test as you go, instead of in huge batches
• merge code to the mainline early and often.
• deploy early and often.
• don't add anything to the system that you don't need until you need it (YAGNI)

Get customer feedback early and often

Get customer feedback early and often to converge on the best possible solution as fast as possible by constantly correcting course instead of blindly following some long-running plan or doing unnecessary work.  The customer's priorities and vision of the product will change as he sees results, so it's better to not find out about these changes after a lot of unnecessary work has been done.

• Do work in priority order as dictated by the customer.  This will allow them to see the aspects that they care about most as early as possible, and therefore help you converge on a unified vision of the product faster.
• Work break-down is better done in vertical slices where the customer can get some amount of real value from each piece (or at least see some evidence of work) rather than having many non-user-facing tasks, where possible.  This allows for more potential for feedback.
• The work shouldn't be considered done by anyone on the team until the customer, or a representative of the customer agrees that it's done.
• Constantly improve estimation skills so that they are generally more reliable so the customer knows the cost of the work he has requested and can make realistic decisions based on that.

Anyway, feel free to try what you want and throw out the rest!

Deadlines have a lot of real plusses:

• The business knows to expect *something* at that time that resembles the original request.  This makes the development process seem somewhat more reliable.
• The programmer gets a sense of clarity about his task when "doneness" can be defined with just a date, and some resemblance to the sliding requirements.
• The programmer gets a sense of urgency, and therefore feels motivated.
• The programmer doesn't have to bother doing the hard work of learning to estimate properly, and doesn't have to feel personally responsible for poor estimates.

I don't mean that with any kind of sarcasm either.  It may sound machiavellian to say things like "deadlines motivate programmers", but the fact of the matter is that they do (they're not the only way however -- I'll get to that later).  In fact, the programmer himself can be the biggest proponent of deadlines, if he has only worked in deadline-driven environments.  That type of programmer will in fact cling to deadlines, because for the aforementioned reasons (people really do want to feel motivated at work).  Without them, he ends up drifting through the day, and not knowing when to stop working on a given task.  He certainly has never learned to estimate properly, and is all-too-happy to never have to bother trying to push-back against management on unrealistic deadlines.

So there are real reasons why deadline-driven development is attractive to all people involved, and it should be no surprise that many software development shops just go from one deadline to the next in a fashion that can only be described as deadline-driven development.  There are certain realities that exist for these shops though, regardless of whether they're recognized:

• Most employees don't want to work beyond their agreed-upon 8 hrs very often.  A system that tries to run counter to this will have high employee turn-over, and therefore incur heavy employee on-boarding costs on a regular basis.
• Technical debt will be incurred constantly.  This is a perilous practice that makes your development process slower over time until the team can no longer develop seemingly basic features in any sort of reasonable time.  It's dangerous because it's very hard to even notice that it is happening until it's too late.

These results create a feedback loop that only serves to amplify the problem:  Employee turnover and on-boarding costs both increase as technical debt increases.  Developers that are new to the codebase tend to create more technical debt as well.  Because of this, the deadline-driven approach to software development can quickly spiral into dead-end codebases that are so mired in technical debt and complexity that no one wants to work on them, and they can't be given new features easily.

Obviously I always go on and on about quality and how it's the limiting factor in being as fast as possible, but there are some necessary trade-offs sometimes when you're developing software in the business world.

For example, when I used to do freelance contract work, I would get a lot of contracts to create one-off microsites for companies that wanted to run online promotional contests.  These microsites would need to go live on a not-too-far-away date where they would run for a few weeks before being closed down and deleted.  I was still concerned about quality, because word-of-mouth is how I got contracts, but technical debt wasn't any concern at all; I knew the software would just be deleted after a month, and I could do manually tested copy/paste spaghetti code to my heart's content if it meant I'd be done faster. *

Also: If I was running a start-up, for instance, I would probably incur more technical debt than if I was working on software that was profitable and had hundreds of users.  It makes more sense in the start-up scenario, because in that scenario, everything is a prototype -- you're not sure if the idea itself is even worthwhile.  You don't need to fight technical debt to ensure a fast future, because you need to be fast NOW, and you might not even have a future.  (Don't get me wrong though… I'd still do a heck of a lot of TDD  and aim for high coverage so I don't have to repeatedly test things manually, but then again I've got a lot of experience with automated testing, and I can write tests quickly enough now that it's an obvious time-saving over manual testing.)

So there are certainly some times to willingly take on technical debt.  There's a great blog post by Steve McConnell on this.

The cold hard facts of the matter though are that if a start-up ends up being successful while taking on technical debt, that technical debt will cost them over time, regardless of whether they realize it or not, or whether they decide to do something about it or not.   It should be obvious that on a long-lived codebase, a team is faster over-all when it minimizes technical debt at every step, rather than letting quality be the dimension that slides as a deadline approaches.  Deadline-driven development is basically a way of borrowing against the future speed of development to meet a certain date. It is tempting because it almost always works over the short-term, but over the long term all the codebases I've seen that are continually developed this way grind the team to a virtual halt at some point.

The corollary to this is that over the long term, a team should theoretically be able to develop faster without deadlines. That will almost certainly seem like insanity to anyone that's only done deadline-driven development, but in my experience it's true; I've worked on such a team, and while it was extremely fast, it was also an easily sustainable pace where everyone was happy with the results.

What does it take to get there?

• Everyone needs to realize that deadlines almost always incur technical debt and that technical debt is a way of borrowing against future development speed.
• Developers need to learn a new definition of done so that their work has a clear point of completion and so that it includes making sure technical debt is not accidentally incurred.
• Developers need to learn how to estimate reliably against this new definition of done.
• Developers need to find a new source of urgency to stay motivated.  (In reality though, meeting their own estimates is usually challenge enough!)
• Everyone needs to acknowledge that estimates come from the developers.
• Everyone needs to figure out how to keep management up to date on the real progress.
• For the most part, management needs to be content with monitoring the speed of progress rather than trying to dictate it.

I'm not trying to say these are easily attainable goals, but I do know that once you've attained them, the team moves very fast.  I basically liken deadline-driven development to a rolling wheel that often stops or goes in reverse briefly compared to a wheel in a more quality-driven approach that may move forward more slowly, but never stops or rolls backward.  In my experience the slower moving wheel that never rolls back has a faster net speed and is more reliable and manageable for the business.

* There's a flip-side to that story too though… After 4 or 5 of those microsites I started to see a pattern in the requirements and started to build up some higher-quality libraries I could use to implement a new site easily to the extent that a site that would take me 3 days would later take me 3 hours.).  You can be sure that I spent a lot of time making that library super DRY and well (unit) tested because that was code that had a future.

I see a lot of people finding their automated tests to be tedious and time-consuming to write and wondering what parts are okay to not test.   A QA manager once asked me a question along the same lines… "How much coverage do you really need?"  I basically answered "What percent do you not care about verifying?"

There are arguments that go the other way:

"Some code is so trivial, that it need not be tested."

Sorry -- I just don't buy it.  One character off in a source file can bring an entire app to its knees.  Any developer that's been coding for any length of time knows this.

"Some code is so low value, that it need not be tested."

Then why are you writing it?  Seriously… if it doesn't matter if it works or not, just delete it.  There's no company that wants you to spend time writing code that no one finds any value in.

"There are diminishing returns as you approach complete coverage, and the effort to do so increases".

I don't buy this one either.  Many people buy into testing the happy paths through the code, but feel they can save time by not testing the error scenarios.  The truth of the matter though is that people are generally pretty lazy about error scenarios in their application code, and there are tonnes of bugs that lie in there as a result.  I usually find the happy path scenarios are well thought out and very few bugs occur there, but the really tricky bits lie in how the code behaves under exceptional and error scenarios.  That can get complicated, and getting it wrong is almost always the same as not handling errors at all.

If you decide that you want to write tests for everything (instead of manually testing everything before each check in), you still have the problem that testing this extremely can be tedious and time-consuming.

The answer is not to test less, but to test smarter:

1. Don't test the same thing twice. If you've tested something to the best of your ability, consider that behaviour tested and don't retest those details in some other kind of test (like an integration test that uses it).  For example, in Rails it's common to find people testing that the built-in validators on their model work, when those validators have been tested to death by the rails core team.  Just test that the proper validators *exist* in your model, and consider that part of the model testing done.
2. Be completely intolerant of boilerplate application code. If you find yourself writing the same stuff over and over, even if it's not pure copy/paste, figure out a way to put that in a library, or a super class or some other form of abstraction.  Then test the hell out of that abstraction, so that every time you use it, you're using something that you don't need to test again.  If you can see repetitive patterns in your application code, even if it's not pure cut and paste, you will have unnecessary repetitive patterns in your test code as well.  Keeping your application code DRY is a great way to make sure your test code is DRY as well.
3. Be completely intolerant of repetitive test code. Learn about how to build more powerful abstractions in your tests as well.  You can...
• • build better matchers (like hamcrest in java or shoulda in ruby)
  • build more powerful assertions, and other helper methods that can be reused
  • use a framework with more configurable 'contexts' than just the usual n-unit 'setup' function
  • use more powerful libraries for setting up test data (like Ruby's FactoryGirl)
  • use the most powerful mocking/stubbing framework you can find for your platform

Test code needs to be DRY too, or it will get painful very quickly.

4. Mock only when you need to. Some people have a tendency to try to mock everything a class uses, but that takes a lot of time, and it makes the tests more fragile.  You also introduce the possibility that the mock doesn't properly mimic the dependency it's supposed to, practically guaranteeing false-positive tests.  You have two other choices:

After having been a freelance developer for about 9 years, I wouldn't recommend it to anyone, but there were a few things I learned in that time that I try to take to my subsequent jobs.

• Learn what "done" means. When you say something's done, it means no more time needs to be spent on it in order for a customer to use it.  People will expect that when you say "done" you're using the same definition as the one in the English dictionary, and not just the works-on-my-machine imaginary definition.
• Be able to justify your technological decisions. The business's primary purpose is making money, NOT making software so even technological decisions ultimately need to have financial justifications.  "Because it's cool" can't be used to justify any technology decision, and generally makes people stop listening to you.
• Make your work process visible, reliable, and honest. The business would almost always prefer that you spend the necessary time to make your process visible and reliable (reliable estimation, task breakdown, etc), than complete sooner.  Don't be that developer that tries to avoid all reporting for a week in the name of saving time, because everyone knows you really just don't like reporting.
• Put effort into all areas of responsibility, not just programming. There are many concerns that a developer must be responsible for other than programming (requirements, specs, testing, deployment, maintenance).  Yes, testing is a large part of the developer's job.  Yes, deployment is a large part of the developer's job.
• Get over the fact that you have to plan and go to meetings. Working with a company requires more rigor, process and communication overhead than working by yourself (meetings, planning, etc).  You should always strive for the lightest possible process that works, but there is always process, even when some would like to pretend that there isn't.  The general rule is that as the number of people increases, so will the number of meetings and the amount of planning that are necessary to coordinate them.
• Don't whine about someone else's code. Everyone already knows that you think that that code that someone else wrote is garbage, because nearly every developer says that about nearly every piece of code he/she did not write.  If it's a real impediment, be specific and constructive.  Offer real solutions other than jumping straight to "rewrite".   A call for a greenfield rewrite is almost always the knee-jerk reaction of the lazy-minded.
• Don't say that something can't be done when it can just to influence engineering or product direction (even when it's really hard). You lose credibility for future consultation quickly that way.
• Constantly improve your estimation skills. Spend the time to break your work down into small enough chunks that they can be reliably estimated.  In the process of working on those chunks, if you learn that your estimate is slipping, report up early and often.  Look back on your failed estimates and figure out what went wrong.  People will only believe your "I can clone google in a weekend" estimates for so long before they decide you don't know how to estimate.
• Don't go to your manager/client with problems unless you're bringing solutions as well (and preferably more than one). You were hired to be the problem solver.
• Keep an eye on your results because over the long haul, results are the only thing that matter. Even if every task you're on fails for some reason that you explain is out of your control, you'll likely be dismissed.  If the reasons really are out of your control, you should really leave first anyway.

One fine day in the trenches, a junior engineer was tasked with optimizing a particular web service call that was way too slow (sometimes slower than 20 seconds!).  I had just figured out how to use a performance profiler for that particular platform, so I thought I'd offer to show him how to use it.  His response was that he was on a tight deadline and didn't have time to learn to use the profiler, and that he was just going to tune the database calls to make them perform better.

Later in the year when the problem still wasn't solved, I had a chance to take it on myself.  The profiler showed me that this service call was itself making around 20 web service calls to a second system that could obviously add up to 20 seconds, if that other web service took 1 second for each of its responses (and the logs showed that sometimes it did).  It turned out that almost all of those web service calls were redundant and completely unnecessary.  Simply removing them (which took about 1 day of effort) shaved 19 seconds off this web service call's duration in production in some cases.

The total execution time spent on database calls was 5 milliseconds.  A 50% improvement there would probably take a week's worth of effort, and really only get you 2 or 3 milliseconds of performance improvement.

The moral of the story is that there's a pretty simple process for optimizing a system:

Skipping step 1 isn't a shortcut.  It's absolutely critical to know that you're optimizing the bottleneck, and not some other random component.

This is pretty obvious stuff for most experienced developers of course.  And while they seem to know to apply this to the systems that they work on, it's more rare that they notice it outside of the code and see it in the process of software engineering in general.

The software development process is a system as well, made up of components.  Requirements come in at one end, and deliverables go out the other end.  As people who are involved in that process, we naturally want it to be more and more efficient.  We are constantly trying to figure out how to optimize that system so we can deliver better software in less time.

So step 1:  Identify and isolate the component that is tying up the most time in the system.

All of these time consumers are affected directly or indirectly by our ability to control quality.  In my experience, the majority of the software development process isn't hung up in requirements analyis, design, or implementation.  It's hung up in all these Quality Control related issues.  Quality Control is the constraint that keeps us from creating the right software faster.  Steve McConnell confirms it in a more scientific way than I ever could:

The rest of us would do well to learn from a discovery made by IBM in the 1970s: Products with the lowest defect counts also have the shortest schedules (Jones 1991). Many organizations currently develop software with defect levels that give them longer schedules than necessary. After surveying about 4000 software projects, Capers Jones reported that poor quality was one of the most common reasons for schedule overruns (1994). He also reported that poor quality is implicated in close to half of all canceled projects. A Software Engineering Institute survey found that more than 60 percent of organizations assessed suffered from inadequate quality assurance (Kitson and Masters 1993). On the curve in Figure 1, those organizations are to the left of the 95-percent-removal line.

(http://www.stevemcconnell.com/articles/art04.htm -- The whole article is extremely illuminating)

What this means is that if we really want to move quickly, we need to take a good hard look at how we manage quality.  Of course this is counter-intuitive, and its actually hard to swallow, because ensuring quality can be mundane and tedious if you're not doing it right.  But as developers we'd be silly to ignore the real bottleneck in the process, and optimizing for that.  Micro-optimizations elsewhere in the process will yield very little benefit compared to improving measures that ensure quality.

As developers, we love to argue about and discuss these micro-optimizations all day long:

It sure seems silly though to be pursuing micro-optimizations, at the expense of tackling the real #1 time-suck:  Quality.

It's no fun to write about stuff that everyone agrees on, so I'm going to write about something that almost everyone seems to disagree with me about:

100% code coverage is almost always possible. It's a valuable target, and it's a worthwhile goal in and of itself.

The primary argument that gets dragged out repeatedly is that 100% code coverage doesn't indicate anything about the quality of your tests whatsoever. It's certainly possible to get code 100% covered without writing a single assert, or by writing incorrect or incomplete tests. If you don't actually assert everything you care about properly, code coverage doesn't seem to be very valuable at all.

While there is a lot of truth to this, unless you have dishonest developers, that code coverage percent does indicate an attempt at quality tests. And really, an honest attempt is all we can ever ask for.

I'm willing to hypothesize though, for the sake of argument that somehow the class under test somehow is 100% covered by wrong tests, incomplete tests, or tests with no asserts whatsoever.  Let's call it an Exercise Suite instead, because that's all it does; it exercises the code.

I'm going to still contend that that coverage metric is a quality indicator. The important thing to notice is that it means you've somehow managed to exercise 100% of your codebase.  This is actually a huge accomplishment that means a few things:

(1) You don't have dead code in your class-under-test.

If you can't exercise every branch in a given class, you've got dead code. Dead code should always be removed to keep from cluttering up future maintenance efforts. (Think you might need it again? That's what version control systems are for -- You can always go back and get it again.)

(2) Your codebase can be loaded into a test harness.

This is actually one of the hardest parts about testing codebases written in procedural languages, or languages that allow scripts, or "includes".  The contents of a given file can't be brought under test reliably and repeatably unless it is possible to load it into the test harness without it actually doing anything until it's exercised by the tests. If you've ever tried to get a legacy PHP project under test, you'll know what I'm talking about here.

(3) You've figured out how to avoid/mitigate side effects.

Code that accesses a database, the filesystem, a web service, hardware, etc is very hard to exercise quickly and without side effects. If your exercise suite has managed to get 100% coverage, while still running fast, reliably and repeatably, you've scored a major victory.

(4) You can detect a myriad of run-time errors.

"Failures" are generally what you have when a test fails. You care about "errors" too though: Errors are all the things that can go wrong at run-time that are obvious just by executing the code with certain inputs. Exercise alone can uncover a number of these. We routinely catch null pointer exceptions in our test suite (attempts to access properties that aren't correctly initialized yet), not through failed tests, but instead simply through exercise. The test runner catches many of them, and always shoves those directly in our face to be solved.

I think my unpopular opinion comes from two things about me that puts me in a bit of a minority as far as developers go:

(1) I've spent a bunch of time actually doing TDD, and I expect unit-tests to look like they were written before the code even if they weren't. If you're writing tests before code, as TDD prescribes, you're generally not going to end up with uncovered code. At any point in the process, the only asymmetry that's likely is that there are tests that are trying to test code that doesn't yet exist. To have uncovered code in TDD generally means you've done something wrong.

(2) I'm extremely accustomed to working with dynamically-typed languages, so I've learned to rely on unit-tests for almost every aspect of verification. I use them to debug. I use them for syntax-validation. I use them to ensure I'm interacting with other objects in ways that they expect (ie, "Did I get that parameter order right in that method call?"). There is no static type-system, so I don't have any quality safe-guard other than the tests I write. In that scenario, uncovered code is a risk I don't want to take -- it could even contain syntax errors!

In short, the hypothetical Exercise Suite has a lot of value in and of itself.

I just finished reading an article on the The Maintainability of Unit Tests and a lot of what was being said there is contrary to my experiences with unit-testing, so I wanted to suggest a few counterpoints.

Basically there's a discussion of how unit-testing can hinder productivity when you have to make major changes to code under test, and more than a few commenters have questioned a goal of 100% coverage.  I'd like to go through those points one-by-one, as well as through some additional points I hear quite often.

"External functional tests would allow more maintenance flexibility under the covers while still finding regressions."

Here's a quote I love to drag around:

“In the industry, people often go back and forth about whether particular tests are unit tests. Is a test really a unit test if it uses another production class? I go back to the two qualities: Does the test run fast? Can it help us localize errors quickly?” (Michael Feathers,  Working Effectively with Legacy Code -- a really great book in my opinion!)

So, unit-tests are simply tests that run fast and help you quickly pinpoint where an error originates from.

External functional tests, on the other hand, are neither fast nor useful at telling you where the problem occurred.  The end result is that you can't run your external functional test suite as you work in small iterations improving your codebase.  You can't run it after every compile and before every check-in, because it takes too long!  And when a functional test suite finds a bug, it can certainly tell you a bug exists, but it's completely useless and showing you where that bug occurred.

"Major changes to a heavily unit-tested portion of code can lead to tedious test changes that take up as much as 85% of the effort"

There are two different scenarios where I've seen this statement applied:

For new code

Instead of doing Test-first development, or test-driven design, our team often postpones writing tests until (1) the requirements have been completely fleshed-out and (2) the approach is known to be a good one.  I think that's perfectly fine -- we're prototyping.

Ideally a prototype would be largely discarded when it's done and TDD would begin for the production-quality code (It would probably be pretty wasteful to not copy/paste large chunks from the prototype code though).  It's critically important that you don't try to prototype your way to production-quality code, so it's best to stop prototyping as soon as the unclearness of the task becomes clear.  After the point that the requirements and the approach are clear, any programming that you do without tests will just be the kind of cowboy-coding you were hoping to avoid.

For code already in production

Any major change should be executed as a series of minor changes.  Every minor change should be preceded by introducing a failing test.  If you follow those two rules, you predictably and systematically meet your goals while guaranteeing that you don't introduce regressions.

It's worth it to take the time to write the kind of code that doesn't need to be revisited over and over in the future, because it was done properly the first time while you had the best understanding of it.

Count yourself lucky that you had all of these existing tests that probably just require minor tweaks.  Take note of how little manual testing you had to do, and how few regressions you introduced.

"It's impossible and fruitless to unit-test view components / UI controls."

It's certainly not impossible.  In our actionscript / flex code, we routinely get 100% coverage on UI controls.  UI controls inevitably have logic and state, and it's not really something we try to avoid.  A checkbox for instance has a state (checked or not) and it has presentation logic to decide whether to show the little black "X" or not.  To move that logic and state somewhere else would be silly and to throw up our hands and say "We can't unit-test UI controls!" would be untrue.  It's true that there are important (visual) functions of that control that still fall outside the realm of unit-testability, but that doesn't mean you can't get that code 100% covered by tests.

"Some code is so trivial that there's no point testing it."

One example given to illustrate this point was a DTO with 20+ properties on it.  What's the point of testing that those properties can be written and read?  It's so trivial that it can't be wrong!

Actually that's not true.  For example: properties always have initial values, and it's easy for a piece of code to rely on a specific default initial value that could be changed by future maintenance.  That's a breakage that static analysis (the compiler) isn't going to know about.  Sure this kind of bug is unlikely to occur in any particular commit, but over time it becomes more and more likely, and your unit tests will always be there to prevent it.

It doesn't matter how trivial a piece of code is: humans can always find ways to get a bug into it.

It's also still very easy to get 100% coverage on that class without writing tests for it at all.  It can be tested indirectly by getting all classes that use it 100% under test.  If you can't get it 100% under test by getting its consumers under test, then it likely has unnecessary features or dead code.  Clean that unnecessary maintenance overhead out and you'll get that 100%.  Is it still a unit-test if you don't have that class directly covered?  Don't worry about that -- ask yourself if that class is covered directly well enough for you to find defects in it quickly.  If it's still not, then we're likely not talking about a trivial piece of code (like a DTO) at all.

"There's a law of diminishing returns as you write more and more minor tests to get 100% coverage."

In practice, we actually see the reverse.  The devil does appear to be in the details:  those things that we thought were minor understandably didn't get much thought during implementation either.  We often don't find bugs in our code (if we're doing post-implementation unit-testing of legacy code) until we actually get down to those minor details.  We're routinely surprised at the issues that we find in the last 20% of our coverage pursuit.

"Unit-testing is not a guarantee of correctness, so we shouldn't get too extreme about it."

I don't want to get too far into explaining the problems with this, but let me just propose the following:

"Ropes can sometimes break and snap, so mountain climbers shouldn't be too strict about using them."

It's absolutely true that unit-testing (even with 100% coverage) does not guarantee correctness.  Anyone who's ever done any significant unit-testing has to wonder at some point "Who tests the test?".  A unit-test is simply a double-check.  If you have bad tests, passing them doesn't mean anything.  Unit-testing is not a silver bullet that will solve all your quality problems: it's just an extremely valuable tool.