I once asked an interviewee “Ever work on a high-performing team?” and he answered “Yeah…” and then went on to explain how his team profiled and optimized their code-base all the time to give it very high performance.

That’s not what I meant, but in his defence, I really don’t think many developers have had the opportunity to work on high-performing teams and so they don’t know what they don’t know.

I’ve been developing software professionally for around 15 years and I’ve only worked on a high-performing team once. It was a couple of years ago, and it was awesome. I haven’t been able to find another since, but now I’m a true-believer in the concept of a 10X team .

I suspect that there might be groups out there somewhere who have experience with even higher-performing teams, but I’m certain that I’ve been on enough teams now that high-performing teams are extremely few and far-between. I’m not sure I even know how to create one, but I know one when I see one. I’ve definitely been on teams since that have been close to being high-performing.

The high-performing team is the ultimate win-win in the software development industry because the developers on the team are thrilled to be going into work on a daily basis and the business is thrilled because they’re producing results much faster and more reliably than other teams. Those are really the two essential criteria, in my opinion:

1. Is everybody happy to be part of the team?
2. Is the team producing results quickly?

The first criteria is absolutely necessary for the second to be sustainable. Any business that neglects developer enjoyment is taking a short-sighted path that will eventually negatively impact performance (due to burnout, employee turnover, etc).

So all of this sounds great, but the real mystery is in how to make it happen. I’m not 100% sure that I know. I don’t think I’ve ever even heard of people talking about how to achieve it.  This is my shot at describing the recipe though.

The Management

The craziest part of my experience is that the only time I’ve seen a high performance team happen was when the manager of the team had very little software development experience. That seems maddeningly counter-intuitive to me because from a developer’s perspective, one of the most frustrating aspects of software development is having a clueless person telling you what to do and how to do it. It turns out that being clueless in software development can be a huge benefit to being brilliant at software development management though, if the manager is already brilliant with management in general.  This manager was definitely brilliant on the management side. Here’s why:

• He hired for attitude over aptitude. He would hire based on peoples’ potential, their attitudes, and their willingness to work hard instead of whether their precise skillset filled a niche that we needed.
• He would cull developers who weren’t team players. It’s not a job anyone wants to do, but it’s sometimes necessary to let people go when they’re not working out. Not doing this results in the rest of the team being miserable.
• He kept the team small, and didn’t move people in or out of it very often. This gives the team the maximum opportunity to form into a cohesive unit.
• He would set clear goals and priorities. No two things would share the same priority (even though I’m sure he was probably secretly flipping a coin when more than one thing was extremely important). That would give the team a clear direction and an ability to focus on what’s most important.
• He never explained a requirement with “because I said so”. This encouraged a culture of questions and feedback which enhanced learning both on the manager’s part and on the team’s part.
• Even when priorities would change, he’d let us complete our current work where possible, so that we wouldn’t have a mess of incomplete work lying around to try to come back to later. I’ve been on teams since where the team has been repeatedly interrupted to the point of not really delivering anything for months.
• He assigned work to the team, and not to individuals, so the team would have to own it collectively. This kept the manager from being the choke-point for “what should I work on next?” questions.
• He didn’t make decisions or solve problems for the team that the team was able to handle themselves (even when individuals on the team would ask).
• He’d bring us the requirements, but leave the solutioning up to us. The result was that the team was largely autonomous and in control of how it accomplished its goals, so we immediately had a newfound sense of responsibility for our work and even pride in it.
• He’d get estimates from us, and not the reverse (ie. deadlines). This way he could inform the business about what the estimated cost and schedule for a given feature was so they could make informed decisions about what features to pursue, rather than just delivering what was finished when the clock ran out.
• He’d actively look for scenarios where the team was not “set up for success” and rectify them.  Nothing is worse for team morale than a death march project, even a mini-one.
• He held the team responsible for the performance of individuals. If someone on the team was struggling on a task (daily stand-up meetings make this pretty obvious) without any help from team, he’d ask the team what they were doing to help.

All of these things created the perfect climate for a high-performing team, but no one person alone (even a great manager) can create a high-performing team on their own.

The Team Member

The individuals on the team were also really impressive to me and no doubt absolutely essential to making the team what it was. Here’s a loose laundry-list of the qualities I witnessed and tried to adhere to myself:

• Members of the team were open, honest, and respectful. They would give each other feedback respectfully and truthfully and would never talk behind each others’ backs. On other teams that I’ve worked on where this wasn’t the practice, team-crushing sub-cliques would form.
• Once a decision has been made by the team, every member of the team would support it (even if he/she was originally opposed) until new evidence is presented. This actually worked well because we would always first take the time to actually listen to and consider everyone’s opinions and ideas.
• Members of the team were results-oriented. “Effort” was never involved in our measure of progress. Software actually delivered according to the team’s standards was our only measure of progress.
• The team would continuously evaluate itself and its practices and try to continuously improve them. We’d have an hour-long meeting every week to talk about how we were doing, and how we could improve (and how we were doing implementing the previous week’s improvements).
• The team would practice collective code ownership, and collective responsibility over all the team’s responsibilities. This is actually a pretty huge deal; if people working on a top priority needed help, they’d ask for it immediately and anyone that could help would. If someone was sick, someone else would take on their current responsibilities automatically. If a build was broken, everyone would drop everything until there was a clear plan to fix it. The morning stand-up meeting would happen at exactly 9 am regardless of whether the manager was there.
• The team would foster cross-functionality as a means of achieving collective code ownership and collective responsibility. What that meant was that members would actively try to teach each other about any particular unique skill-sets they had so that they would be redundant and team members would be better able to take on any task in the work queue.
• The team would be as collaborative as possible. We all sat in a bullpen-like environment where we’d torn down the dividers that were originally erected between us. We’d often pair-program, not only for skill-sharing and mentoring, but also for normal coding tasks, because the boring tasks are less mind-numbing that way and the complicated tasks get better solutions that way.

In the 5 or 6 teams in our organization at the time, I think our team was by far the fastest, and most capable of handling the more technically diverse and challenging work. No one on the team was a rock-star programmer, but somehow despite that, the team as a whole was really outstanding.

Eventually when this manager moved on, I took the management role and ultimately failed to maintain that environment (partially for a number of reasons that were out of my control, and partially because at the time I didn’t realize the necessary ingredients).  I’ve only really learned them since then by watching dysfunctional teams.

It’s a cliche, but in the right environment, the collective can indeed be greater than the sum of its parts.

Suggested Hacker News title: “Show HN: I’m only 37 and I wrote this over the weekend (plus the previous ~400 days)”

First a little background

I’ve always been a big fan of REST the way Roy intended it for two major reasons:

• It uses HTTP the way HTTP was intended, playing on its strengths and not wasting any time re-inventing what HTTP already has.
• The state of HTTP APIs these days is really ridiculously bad, given what we should have learned from the unfathomable success of the WWW.

I was also envious as hell at the great framework Erlang developers had in Web Machine .  If you write HTTP APIs for a living and you haven’t seen HTTP mapped to a state machine, you really owe it to yourself to check it out.  It’s brilliant.

The thing I really wanted to do though was make web APIs not so terrible.  There are a few reasons that I think they’re terrible:

• You have to read reams of documentation to do the simplest things, like finding the URL for something.
• You have to hard-code urls all over your client app.
• When you do something wrong, there’s no consistency in error output.
• Knowing HTTP doesn’t make it any easier to predict how something will work.

If you rely on someone else’s API in your day-to-day work, you probably know what I mean.  I’m not going to name any names (but here’s a clue: one of the worst offenders starts with an F and rhymes with Facebook).

So the APIs I wanted to make and to use would be:

• semantically consistent with plain old HTTP
• discoverable, like other pages and controls on normal HTML web sites are discoverable.

Luckily HTTP is also the vanguard of API discoverability, most often in the form of hypertext (So that’s what that H stands for!) .   I’m a bit blown away that this wasn’t obvious to me earlier, but luckily I stumbled into Roy Fielding talking about hypermedia and I realized that that’s precisely the correct way to solve the discoverability problem.  He was saying real REST APIs should minimize out-of-band knowledge by leaning on what HTTP’s already got: links! * .

Then I saw Jon Moore make it real with an XHTML API .  XHTML forms are a method of discoverability as well?!? My mind was blown.  Think about that: XHTML documents tell you what kind of write-possibilities the API has.  You don’t need to read docs to find out!

And then I saw Mike Amundsen bring it all back to json   and I knew it might be something I could actually use in the real world and people trying to Get Stuff Done wouldn’t want to hunt me down and murder me in my bed (especially since I’m also a card-carrying member of the People Who Want To Get Stuff Done Liberation Front).   I think most of us are of the belief that XHTML isn’t going to unseat json as the preferred media-type for APIs anytime soon.

Armed with all this inspiration, I wrote a web-framework.  I ended up picking node.js mostly because the HTTP line-protocol stuff was already covered there without any extra cruft to get in my way, so I was free to try to aim for “Web Machine with json with links”**, which I call Percolator.  The framework isn’t the interesting part though; it’s the stuff I learned along the way, which can be done in any framework/language/platform, even if it’s not baked into the framework.

The First Interesting Part:  Hypermedia apis are indeed awesome

I started by just putting links in my json to other endpoints that make sense.  That made my json APIs so much easier to use, even when the user is me.  I had one endpoint for my entire API and every other endpoint was discoverable by following links.  I used JSONView for Chrome and for Firefox and I could click around my entire API testing the GET method on every endpoint just by clicking.  Here’s how it looks:

Those purple links are visited links that you can actually click on and get a new API response.  I showed some co-workers that were using my API and eventually they stopped asking “Hey what’s the URL for getting X again???”  They could always easily re-find it themselves.

Adding links also pays off when humans aren’t involved.  In many cases it means that the client doesn’t need to construct its own links.  For example: when one of my APIs was being used for replication by a client app the client developer asked how we’d handle pagination, because he wanted to page through all my results to replicate my API’s data elsewhere (sort of like how couchdb replication works, which is simple yet amazing ) .  I had a pagination scheme working, but it didn’t make use of the database indices as well as I wanted so I told him instead to just follow the ‘next’ link and not to worry about the parameters for constructing the URL at all and that the absence of a ‘next’ link would indicate the end of the results.  What this meant for me is that when I changed the pagination scheme to something more friendly to my database, I didn’t even have to tell him.  What it meant for him is that he never had to change anything on his side to support the new form of pagination, and he didn’t have to construct any urls.

It blows my mind that pushing this simple technology further could end up with clients ONLY needing to store the root URL of the API, and not a who’s-who of possible disparate endpoints that practically mirrors the routing table on the server.    People are actually starting to push the envelope this way.  Wouldn’t it be great if at some point we could stop writing client software that’s hopelessly service-dependent ?  Wouldn’t it be great if one day client-side url construction was considered as silly as using regexes to get data from json?

I should also add that even if you’re writing both the client and the server, which happens a lot with single-page apps and mobile apps, you might be tempted to think this is a waste of time, but in my experience it makes testing and debugging much easier and helps me remember what my API does after a month without using it despite my terrible memory.  That minor extra effort of adding links pays for itself over and over again.

Interesting Part #2: JsonSchema is surprisingly awesome

Of course once I could surf around my API, it didn’t take long for me to bemoan the fact that I could only peruse GETs this way.  I had seen Jon Moore’s XHTML API demo  (This is my third link to it here, so you know it’s worth a watch!) , and I was jealous of how XHTML forms could describe the POSTs as well.  Originally I decided that json needed forms as well, but the big brains in the Hypermedia-Web Google Group  convinced me that I could just expand my link concept to know about other HTTP methods.   And I had already stumbled upon JsonSchema as a way to specify what was expected in a PUT or POST payload, just like XHTML forms can do (but still using json of course).

JsonSchema is a particularly elegant solution to the problem, because I can write all my validations in JsonSchema, publish the validations as json to the api (JsonSchema is just json afterall), and then on the client side re-use them for validations before I even bother with an http request to the api.  This way I can use the same validations on the client and server, AND communicate them naturally as part of the json payload.  The end result was that my json APIs could now fully communicate everything that was possible via the json API itself.  Here’s how it looks:

I added baked-in JsonSchema support to Percolator right away of course, but you can do it easily in any of the most popular languages  ( see the validators tab).  Currently I haven’t implemented any clients that use it for validations yet, but I do use it for my server-side validations.  More importantly still, it communicates to humans everything that’s necessary for a successful PUT or POST. ***

Ultimately the addition of JsonSchema lead me to add a static frontend to Percolator that, after abstracting it away, I published separately as the Hyper+JSON Browser .  You don’t need it, but you can use it in conjunction with Percolator apps to ‘browse’ them even more easily than JSONView does (and without installing a plugin).  Any app (using any language/framework) that spits out the same sort of output as Percolator could use this the Hyper+Json Browser just the same, because it’s a static single-page app that works over any API that works similarly.  Here’s how it looks in use:

When you click on that ‘create’ link to POST, it does this:

That red json logo indicates that you haven’t entered valid json (yet).  Pressing OK will actually POST the json that you enter and show you the result.

Here’s a view of another resource (an individual item in the list from the previous collection resource, which can be found by clicking that item’s ‘self‘ link.  It shows delete and update links for this particular resource.

People have likened this to the Django admin interface, which is a huge compliment, not only because the Django admin interface is pretty legendary, but also because this frontend doesn’t know a single thing about the API other than the requirement that the API uses the aforementioned link format.  This is an area where I’m really excited: In what other ways can other programs use these APIs without foreknowledge?  (I’m not really talking about intelligent agents  – they’d need to understand a little more than the API provides — but dumber things that still use human interaction eventually, like automatic HTML form generation, are mostly possible.)

(Somewhat less) Interesting Part #3:  Object-oriented routing leads to less stupid stuff

I hate to sound old-school, but sometimes the object-oriented paradigm maps to a problem better than a functional one.  In this case, I’m talking about routing.  I don’t know why the HTTP methods GET, POST, PUT, etc are named “methods” but it sure does make perfect sense, because each one is tied to a particular “resource”: that “thing” you’re interacting with by using the HTTP methods for a given URL.  It should be obvious then that HTTP resources should be the thing we route to, and not disparate methods.  REST APIs are supposed to be the antithesis of RPC, but when routing involves the method as well, developers tend to make less RESTful APIs and framework developers tend to mess up some of the finer points of HTTP.  Here’s a quick test:

• Grab your favorite web framework and create a resource with a single GET handler.
• POST to it.  Did you get the proper 405 response?
• send an OPTIONS request.  Did it tell you that only GET was available?

Probably not.  Web-machine gets this right because it routes to the resource, instead of disparate functions that when imagined together form a resource.  I wrote Percolator to work like web machine in this way, because I wanted my APIs to look like something other than a haphazard collection of RPC calls without requiring a lot of self-discipline.  The result is that Percolator responds with 405s when a client tries a method that that resource doesn’t support, and also responds to OPTIONS requests with a list of all the allowed methods.  The result again is better discoverability without reams of documentation, and better feedback for client developers when they’ve done something the API doesn’t support.

There are advantages to OO routes for application developers to though, organization-wise:

• All routes for a particular resource are connected to that resource, so there’s no chance of sloppy intermingling in the source code.
• A bunch of code that you have to write in each method can be written just once for the resource.
• You start to see patterns in what resources do, which can lead to certain classes of resources that can be re-used over and over, sometimes wholesale and sometimes with particular strategies that change them slightly.  This is a bit of a big deal, in my opinion because it’s extremely rare that I see this kind of re-use in web applications in the real world.

 

Anyway:  Percolator is indeed ready for production use (at least for the use-cases for which I’ve used it in production!).   It’s extremely fast and extensible, and I hate it a lot less than anything else I’ve used for json REST APIs.

Hopefully it helps someone make a nice API, or it gives some other framework developer some new ideas.  The APIs we have today cannot possibly be the best that we can do.

* plus a pile of other conventions
** Ultimately I diverged quite a bit from Web Machine, but it was always a huge inspiration.
*** Alright not everything, but a heck of a lot.  Linked documentation on my RELs would be another huge boost to discoverability.

I’ve seen these anti-patterns in a lot of web APIs, so I thought I’d detail why they’re bad and what to do instead.

Using a 200 status code for every response:

This anti-pattern is common for SOAP-like RPC APIs.  Using a 200 status code for all your responses is silly because you end up having to reinvent an error-indication in your response body instead, and nobody in the world knows your new standard (or wants to have to learn it).

It’s also extremely confusing in the cases that an error has occurred because a status code of 200 means everything is A-OK.  There are already a bunch of error codes in the 400 and 500 ranges that are fully capable of indicating all sorts of common problems. It’s still great to put an error message in the response body, but there should be a 400 or 500-level status code so people know that an error even occurred and don’t go ahead and try to interpret the response body as normal or otherwise assume a success.

Using a 500 status code for all errors:

This anti-pattern is common because 500 is what most frameworks will do when a web application throws an uncaught exception, and so it becomes convenient to just throw exceptions and not worry about the repercussions. This is mildly better than using a 200 status code for literally everything, because at least it shows an error occurred but it still causes a lot of confusion for client developers.

When you 500 for an error (even if your web framework did it for you), you’re signalling that the problem with the request was the fault of your application (the server) and it’s possible that retrying the request will be successful. Because 500 errors indicate an internal server error, they shouldn’t actually ever happen though* — In general, you’ve got a bug in your application if they do, and the web server will tell everyone about it. The right thing to do, if there’s something wrong with the request is to respond with the appropriate 400-level status code.

Using a 500 error when a 400-level status code is more appropriate is obviously really confusing to the person making the requests because it doesn’t even let them know that their request needs to be fixed. Even if that person is also you, it forces you to start digging into your back-end for what could possibly just be a bad client request. If you don’t know which 400-level code to use, at the very least use 400 (Bad Request) itself. This will at least give your client developer an indication if the blame is on the server or the client and cut your debugging in half. A few words about the error in the response body can go a long way as well.

Logging 400-level codes at ‘error’ level:

Logging is pretty important to measuring the quality of implementation of an API, and learning about specific defects. The ‘error’ level in your logs should be a way to indicate “things that should not have happened that are within this application’s control”, so that you actually have a hope of attempting a 0-error policy. 400 level codes are errors of course, but they’re client errors, and not your server application’s error, so they should not be logged as errors along with your actual 500-level errors (Of course you might still log them as “info” level to keep track of what kinds of crazy things your clients are doing).

Why do all these details matter?

When you have an interface between two applications (the client and the server in this case) like HTTP, it REALLY simplifies things a lot if they communicate whether messages are acceptable or not (and why). Even if you control both ends of the interaction, you can save yourself a lot of server-side debugging by having the server indicate what exactly went wrong with a given request, instead of repeatedly reading stack traces every time some JSON you POSTed is missing a property. Compared to how little extra work there is involved in communicating the problem correctly, it’s pretty short-sighted to just let all exceptions bubble-up, or catch all exceptions silently (and 200).

Keep in mind too that if you’re now convinced that communicating errors is important, then HTTP already has a system for just that: HTTP status codes (plus whatever specifics you want to additionally communicate in the response body). Status codes are part of the greater HTTP standard that a lot of developers, libraries, proxies, and caches already understand. Re-inventing your own error mechanism is generally a waste of everyone’s time (with you at the top of the list).

Getting this right is a real time-saver over the long-haul even if you’re the only consumer of the API writing both client-side and server-side.  The time-savings only multiplies as you add more clients.

* Sometimes 500 is appropriate of course: If your application requires a database and its lost connectivity to it, then a 500 makes total sense. It signals that there’s a problem in your application and the client can possibly try the request again later.

If you consider the number of packages on pypi, npm, and rubygems, and the release dates of python (1991), ruby (1995) and node.js (2009), it looks a lot like those communities are getting new packages at the following rates, per year:

python:  29,720 packages / 22 years = 1351 packages per year

ruby:      54,385 packages / 18 years =    3022 packages per year

node.js  26,966 packages / 4 years =   6742 packages per year

It’s important to note that I’m only showing the other languages as a measuring stick.  There are probably a lot of reasons for the disparity here (including my imprecise math), and I’m not trying to say anything like “node.js > ruby > python”  but obviously new node.js packages are being published to npm at a staggering rate.   I just checked npm (on a Sunday night) and 20 new versions of packages have been published to npm in the last hour alone.

This is a pretty amazing feat for node.js.  How is this possible?

Batteries NOT included

If you’ve needed an HTTP client in Python, you’ve probably asked yourself “should I use urllib, urllib2, or httplib for this?” like so many people before you.  Well the answer is probably that you should use requests.  It’s just a really simple, intuitive HTTP client library that wraps up a lot of weirdness and bugs from the standard libraries, but it’s not a standard library like the others.

The “Batteries included” philosophy of Python was definitely the right approach during the mid 90’s and one of the reasons that I loved Python so much; this was a time before modern package management, and before it was easy to find and install community-created libraries.  Nowadays though I think it’s counter-productive.  Developers in the community rarely want to bother trying to compete with the standard library, so people are less likely to try to write libraries that improve upon it.

Developers are less likely to use non-standard libraries in their projects too.  I’ve heard many instance of people suffering through using an inferior standard library just to have “no dependencies”.  But in this day and age of cheap massive storage and modern package management, there are very few reasons for a policy of “no dependencies”.

Conversely, the node.js core developers seem to actually want to minimize the standard library as much as possible.  They have repeatedly removed features from the standard library to free them up to be implemented by the community instead.  This allows for the most variety and lets the best implementation win (but only until someone makes a better one, of course!).

Imagine how liberating this is for standard library maintainers too.  The node.js standard library is comparatively tiny, freeing the core developers to just deal with the core necessities.

The Tiny Module Aesthetic

Just like the 140 character limit on twitter makes people “blog” more, the node.js community has a culture of publishing tiny modules that makes people more comfortable with publishing smaller packages, and so it happens vastly more often.

While I don’t really consider myself “a node.js developer”, I’ve found myself publishing over a dozen packages in the last year myself, just because I’m working on a node.js project at work, and I publish any reusable package from that work that I create.  My project ends up factored better, and I end up with a bunch of building blocks I can re-use elsewhere (and then I do!).  Sometimes random people from the interwebs even fix my bugs before I notice them!

This approach is not unusual at all either, at least in the node.js community.  The community is lead by developers that have each published literally hundreds of packages.  I don’t even know how this is possible.

Of course there’s absolutely nothing to stop other languages from following suit, so it seems to me like community culture is the deciding factor.

Massive monolithic frameworks like Ruby on Rails tend to not take hold in the node.js community mindshare, I think due in part to this “tiny module aesthetic”, and it’s all the better for it.  It seems to me that monolithic frameworks like Rails don’t have a clear notion of what does not belong in Rails, and so they tend to expand indefinitely to include everything an application could need.  This leads to people trying to assert that Rails is the best solution for all problems, and having a cooling effect on innovation within the web application space in the Ruby community.

The tiny module aesthetic also leads to an ecosystem of extreme re-use.  When a package doesn’t do more than is absolutely necessary, it’s very easy to understand and to integrate into other applications.  Conversely, while massive monolithic frameworks tend to appear in a lot of end-user applications, they rarely never end up being dependencies in other packages themselves.  It seems unfortunate to me that so much great software should be so difficult to re-use.

git and github.com monoculture

To be honest, few things are more boring to me in software development than version control, and I find the complexity of git to be a bit silly and mostly unnecessary for me, but I’ve never heard of a node.js developer that doesn’t use it, and there’s huge value in that kind of monoculture.  It means that they all speak with a common language and there’s no impedance when you want to contribute to someone else’s projects.

Github.com has a similar effect in lowering the barriers for cross-pollination between developers.  I’ve only very rarely seen a node.js project that wasn’t on github.  This means I immediately know exactly where to find source code if I want to contribute.

The pluses and minuses of a github monoculture go far beyond this, but there’s probably enough fodder there for an entirely different post.

Permissive Licensing

Node.js packages tend to use extremely permissive licensing like MIT and BSD licensing.  In fact, the default license for a package created by `npm init` is BSD.  I think this is another sign of the times.

Very few people care anymore if someone else forks and doesn’t contribute back.  There’s very little to gain in not contributing back anyway because of the cost of maintaining your own fork.

This is important because no one wants to have to deal with dependencies that make them think about legal repercussions, or give them extra responsibilities.  Licenses like the GPL end up having a cooling effect on software-reuse for this exact reason.

If you’ve developed SaaS in a large organization (or even a small organization that acts like one), you’ve probably seen some of the symptoms of fear of releasing software:

• Off-hours deployments and weekend deployments
• Infrequent releases
• extended code-freeze and testing cycles
• Reluctance to refactor
• Feature branches for the purpose of ‘cherry-picking’
• release managers
• many non-customer-facing environments before production.

These symptoms usually happen only in larger/older organizations, because of two factors:

• They take a lot more personnel / time to implement than smaller organizations have.
• They are usually processes and attitudes that are collected as a reaction to problems that occur in the course of a company’s lifetime.

“But you can’t be too careful!”

One problem is that you CAN indeed be too careful. It’s absolutely imperative that the costs be considered as well, if you want software development to still go reasonably fast. Otherwise these manual preventative processes will slowly creep in over time and your team will be releasing at a snail’s pace.

The other problem is that these policies alone are in some ways not careful enough. Where are the techniques for mitigating defects that inevitably still slip through? Where are the techniques to detect them, limit the customers exposed, and limit the time that customers are exposed?

All of these symptoms have a few things in common: They tend toward heavy manual and preventative process added generously over time, usually with little thought to the cost/benefit ratio of such process. It’s all too common for every production issue to result in a new process to prevent it, without regard to how infrequently that problem might occur, how damaging the result is, or how easily it is rectified.

So how do we keep bugs from getting to our customers?!?

Non-preventative measures (I call them “Reactive Measures“) do indeed exist, and can help you safely maintain a fast and steady flow of new features to customers.

• Feature-Flags allow you to commit incomplete work to master and push it to production safely. Customer-specific feature flags allow you to test in production or enroll customers in a beta program. With a good feature-flag system, it’s even possible to release a feature to only a certain percentage of customers, and to watch your metrics to see the effects.
• Easy/fast (read “Automated”) deployment. If deployment takes a long time, then you can’t afford to deploy often. Also, if you can’t deploy quickly, you can’t redeploy quickly (in the event of a problem).
• Easy-undo. Rolling back from a defective release to a known-good release is the simplest and fastest way to recover from defects, if the process is both quick and easy. Defects will happen no matter how much prevention you use: Easy-undo can mean the difference between light customer impact and heavy customer impact.
• Smaller releases. Smaller releases have less code, and therefore generally less bugs. Because they don’t contain many changes, they’re easier to test and a rollback is less-likely to negatively impact a customer. Also when a production defect appears in the latest release you have much less code to sift through to determine the cause.
• Practice Continuous Integration (which necessarily involves always committing to master, and only incidentally has to do with using a CI server) . Feature branches, infrequent commits, and committing non-production-ready work are all practices that need to be avoided, because they are all time-consuming delaying mechanisms.
• “Done” == “In Production”. If the team moves on to new features before the last one is in production, then that last feature is obviously unnecessarily being held back from release. They will be faster if they deliver to production and ensure it functions there properly before moving on to new features because it avoids unnecessary and time-consuming multi-tasking. This goes hand-in-hand with smaller, more frequent releases.
• Comprehensive Automated testing. When you deploy more often, you need to test more often. The cheapest way to do this is with comprehensive automated tests. While they’re a heavy-upfront cost, if you run them as often as possible, they’ll pay for themselves very shortly.
• Continuous Production testing. Create automated *black-box* integration tests for some of your most important features and figure out a way to run them repeatedly and continuously against your production deployment so that you know instantly when those features are broken. There’s no need to wait for customer feedback to learn about a faulty release.
• Searchable, aggregated logging with proper log-levels. Making your logs easily searchable in one place makes them something that developers will actually use for defect detection and for debugging. They can go a long way to making your application more transparent.
• Runtime monitoring, runtime metrics and information radiators (most importantly alerts). Instrument the things you care about most, and have alerts sent out when the measurements are abnormal.

Note: Of course some preventative processes are necessary, but when they are, automation is almost always preferable to time-consuming manual human processes.  Either way, you should consider the cost of the process and determine if it the fall-out from the type of defect it prevents is actually worth that cost.

In the end, a fear of releasing is a fear of change, and a fear of change is a fear of improvement. This is a crucial lesson that Big Company software teams can learn from the small, scrappy start-ups that are quickly trying to disrupt them. The more often you release, the more often you can test hypotheses and get customer feedback, and the more often you can course-correct. Companies that don’t realize this are susceptible to being leap-frogged by competitors that do.

I’ve been doing a lot of work with node.js recently, and since I care about testing and test coverage quite a bit, I’ve had to solve the problem of how to test classes that have what I’d call “fake-worthy” dependencies like database clients, or the file system, or network i/o (to name only a few). When I call a dependency “fake-worthy”, I mean that it’s often easier/better/faster to use a fake version of it in the tests than the real thing for a number of reasons that I won’t try to get into here.

I’ve used IOC containers on other platforms, and I’ve seen a number of ways to make require() do different things depending on whether you’re in a test environment or not. In the end, I’ve decided to just follow a simple pattern that is extremely low-tech compared to anything else I’ve seen. It should be usable in browser javascript just the same.  Here’s the skinny:

TIP #1: Pass in any fake-worthy dependencies as optional parameters.

Yes, I will change application code to make it more easily testable.
For example, if I want to test this code:

function addUserToDatabase(user){
 var db = new Database('127.0.0.1', 'someapp');
 db.insert('users', user);
}

I’ll want to test that addUserToDatabase() does its database work correctly, but I won’t want my test suite to actually hit the database:

It could instead be written as:

function addUserToDatabase(user, db){
 db = db || new Database('127.0.0.1', 'someapp');
 db.insert('users', user);
}

This let’s me do:

addUserToDatabase({name : "Joe"});

…which instantiates a new database object, because when you don’t pass a db, one gets created for you. But it also lets me do :

var fakeDB = {
 insert : function(table, obj){
   assert.equal(table, 'users');
   assert.equal(obj.name, 'Joe');
 }
}
// inject fakeDB via the second param
addUserToDatabase({name : 'Joe'}, fakeDb);

…which lets me pass in a fake database object for the purpose of testing that addUserToDatabase() calls the database correctly.

TIP #2: Don’t do anything fake-worthy in a constructor

It’s often tempting to do all kinds of initialization in a constructor so that the object is ready to use by all methods right after construction, but that can lead to problems.

For example, this code:

var CarModel = function(){
 this._db = new Database('127.0.0.1', 'someapp');
 this._db.connect(); // makes a network call to the db
}
CarModel.prototype.add = function(car){
 this._db.insert('cars', car);
}

If you don’t want to use the optional parameters trick for this constructor, you have a second option because you’re being object-oriented. You can simply call connect() outside the constructor:

var CarModel = function(){
 this._db = new Database('127.0.0.1', 'someapp');
}
CarModel.prototype.add = function(car){
 this._db.connect(); // makes a network call to the db
 this._db.insert('cars', car);
}

This makes it possible to inject a dependency using a property of the object, without changing any function signatures. Here’s an example test doing just that:

var connectWasCalled = false;
var insertWasCalled = false;
var fakeDB = {
 insert : function(table, obj){
   assert.equal(table, 'cars');
   assert.equal(car.name, 'Tesla Model S');
   insertWasCalled = true;
 },
 connect : function(){
   connectWasCalled = true;
 }
}
var carModel = new CarModel();
carModel._db = fakeDB;  // injection!
carModel.add({name : 'Tesla Model S'});
assert(connectWasCalled);
assert(insertWasCalled);

TIP #3 Use Lazily Initialization Methods

Removing fake-worthy activities from the constructor is what I’d call “Lazy Initialization”, meaning that you don’t get the dependency ready for use (db.connect() in our example) until the last reasonable moment. When you want to have a dependency lazily initialized for other methods as well, it can help to create a lazy-initialization method instead. Imagine we add a new ‘remove()’ method to our CarModel:

var CarModel = function(){
 this._db = new Database('127.0.0.1', 'someapp');
}
CarModel.prototype.add = function(car){
 this._db.connect(); // makes a network call to the db
 this._db.insert('cars', car);
}
CarModel.prototype.remove = function(name){
 this._db.connect(); // makes a network call to the db
 this._db.remove('cars', name);
}

What I would do here instead is refactor to this:

var CarModel = function(){
 this._db = null; // just to be explicit
}
CarModel.prototype.db = function(){
 if (!this._db){
   this._db = new Database('127.0.0.1', 'someapp');
   this._db.connect();
 }
 return this._db;
}
CarModel.prototype.add = function(car){
 this.db().insert('cars', car);
}
CarModel.prototype.remove = function(name){
 this.db().remove('cars', name);
}

To test add() in this scenario, I’d just substitute the _db property with our fake:

var connectWasCalled = false;
var insertWasCalled = false;
var fakeDB = {
   insert : function(table, obj){
   assert.equal(table, 'cars');
   assert.equal(car.name, 'Tesla Model S');
   insertWasCalled = true;
 }
}
var carModel = new CarModel();
carModel._db = fakeDB;  // injection!
carModel.add({name : 'Tesla Model S'});
assert(connectWasCalled);
assert(insertWasCalled);

These three patterns have made it so that I can easily deal with any fake-worthy dependencies by either monkey-patching them in or passing them as optional arguments. It doesn’t feel as clean as it could be at first because:

• I generally try to avoid monkey-patching.
• I usually try to avoid test-only use-cases in my application code.

I’ve decided to trade those two values away for simple testability though. While other people can wrestle with mocking frameworks that do strange things with require(), or try to devise an IoC container for javascript, I think I’ve found a methodology that won’t actually add as much complexity in the long run.

Because Javascript objects leave their internals open for access (they have no private/public access modifiers), I consider mucking with the internals to be allowed for testing purposes. I generally consider it allowed for any purpose really, because we’re all adults here, but of course for other purposes, the developer needs to know he does so at his own risk.

Once you embrace this sort of philosophy, you realize Javascript has a built-in DI framework.

Alright… here is my initial idea for a hypermedia format based in JSON, that I’ll probably now call Hyper+JSON, because if it gets any type of official approval it would probably get a mediatype name like application/vnd.hyper+JSON .

A lot of the justifications for the design can be found in here and here. Basically I want to create a JSON format with links and forms.

I think there are two basic trade-offs that I made that could be controversial.

1. I’ve seen links implemented in an array in a few other types, but I think implementing them as a set of name/value pairs where the name is the “rel” attribute is more in the spirit of JSON, and easier to target programmatically in an unserialized object.
2. Instead of doing anything like HTML’s form elements, I think a JSON form should just contain a JSON schema object. JSON forms don’t need to describe UI elements — they just need to describe the JSON that the client should be sending. JSON Schemas are also interesting in that you get a chance to specify validations better than HTML forms currently can.

So here’s the initial description:

Hyper+JSON is a simple hypermedia format that is also valid application/json. It is designed to add links, forms and uri-templates to JSON. It is similar to XHTML / HTML in the use of links and forms for navigation and state change purposes, and additionally adds uri-templates that allow parametric construction of new urls by a client. A Hyper+JSON form differs semantically from an HTML form in that it simply contains a single JSON schema document specifying the schema for any JSON object to be sent. Like HTML though, it is wrapped in a ‘form’ element that has ‘method’ and ‘action’ parameters.

application/vnd.hyper+JSON

All documents related to the Hyper+JSON mediatype shall use the keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” to describe requirements. These keywords are to be interpreted as described in [RFC2119].

General Concepts:

While JSON is one of the most popular formats for data interchange over HTTP APIs, it is impossible to use it as a format for hypermedia APIs because it lacks hypermedia controls (links and forms). Hyper+JSON simply aims to add links and forms to the JSON format in a standardized way so that Hypermedia APIs can be written in JSON.

Hyper+JSON recognizes 2 object types that are not specified by JSON: the “_links” object and the “_forms” object. JSON does not provide a means for defining JSON object ‘sub-types’, but in a Hyper+JSON document, an object or a property of an object with one of these names MUST be considered to be a _links or _forms object as defined in this specification.

1.1 The _forms object

The _forms object MAY be added as a property to any JSON object (a root object, or any number of sub-objects) to represent a list of available “forms” as name-value pairs, where the name is the form’s name, and the value is the form specification itself. A Hyper+JSON form is similar in nature to an HTML form but unlike HTML forms, it is completely unconcerned with presentation-oriented elements like buttons, labels, etc. It is only interested in conveying the necessary information for validating and writing data to the server. That necessary information includes the URI, the method, and a specification of what form of JSON is acceptable, detailed by a JSON Schema document.

Here’s an example usage:

{
  "musicians" : [
    {"firstname" : "Joe", "lastname" : "Strummer"},
    {"firstname" : "Neil", "lastname" : "Young"},
  ],
  "_forms" :  {
    "add-musician" : { "href" : "/musicians/",
              "method" : "POST",
              "schema" : {
                 "description" : "A musician",
                 "type" : "object",
                 "properties" : {
                   "firstname" : { "type" : "string",
                                   "title" : "First Name"},
                   "lastname" : { "type" : "string",
                                   "title" : "Last Name"}
                 }
              }
     }
  }
}

In this example, there is a single _forms object, describing a single form named “add-musician”. That form is in turn another object that contains REQUIRED “href”, and “method” properties as well as an OPTIONAL “schema” property.

1.1.1 The href property

The href property is a REQUIRED property of a form object that exists to tell the client what URI a form request should use. The form object MUST have an href property. The href property MUST contain a valid URI. This URI MUST represent the address used to send the form request.

1.1.2 The method property

The method property is a REQUIRED property of a form object that exists to tell the client what HTTP method to use for the form request. The form object MUST have a method property. The method property MUST contain a valid HTTP method. The method MUST be allowed for the resource at the URI specified in the HREF property.

1.1.3 The schema property

For forms where the HTTP method expects an accompanying HTTP body, a schema SHOULD be provided and MUST be a valid JSON Schema document. The schema MAY be used for validation purposes both on the client and the server, though any given request may be subject to additional validations not described in the schema.

For form requests that expect an accompanying body, the client MUST send that body with an application/json content-type and content-type header.

1.1.4 Hyper+JSON forms as URI-templates

In the case of HTTP GET requests described by a form, the schema property MAY still be included, in order to describe parameters that can be added to the URI’s querystring component (rather than as a JSON body). In this case the schema must describe a simple JSON object with only strings and number values (no nested objects or arrays) so that it can be serialized successfully as a query string. HTTP GET requests SHOULD NOT include the application/json content-type, as they SHOULD NOT have a body.

2.1 The _links object.

While Hyper+JSON forms can make body-less GET requests to static urls, the _links object can be a more convenient and natural way of doing so.

The _links object MAY be added as a property to any JSON object (a root object, or any number of sub-objects) to represent a list of available “links” as name-value pairs, where the name is the relationship name (often specified by a “rel” attribute in HTML), and the value is an object that SHOULD contain an “href” property with a value containing the URI of the link object. A Hyper+JSON link is similar in nature to an HTML link  but unlike HTML links, its “rel” property is the name for it in the _links object.

Here’s an example usage:

{
  "musicians" : [
                {"firstname" : "Joe",
                 "lastname" : "Strummer",
                 "_links" : {
                     "self" : {"href" : "/musicians/1234"}
                 }
                },
                {"firstname" : "Neil",
                 "lastname" : "Young",
                 "_links" : {
                      "self" : {"href" : "/musicians/1235"}
                 }
                }
  ],
  "_links" :  {
    "self" : { "href" : "/musicians/"},
    "parent" : {"href" : "/"}
  }
}

2.1.1 The href property

The href property is a REQUIRED property of a link object that exists to tell the client what URI a request should use. The href property MUST contain a valid URI.

That’s it so far

I really don’t want to add anything else really, because the main attraction to JSON is simplicity, and I’m certain that this will already be quite a bit more complex than many developers will wish to support.

I have no idea what the next steps are to getting this more officially approved, but I’ll be looking into it for sure.

One thing I’d really love is some feedback, because I could be missing something that already exists, or missing other great opportunities to make this more powerful, or even more simple.  I don’t even know what venues to look for feedback in, but I’ll give that a try too.

I wrote about this once before when I was just starting to “get it”.

I want there to be a json sub-format that has all the awesomeness of hypertext.

Hypertext? What’s that?

Well HTML and XHTML are probably the most common, but also RSS and atom and probably more that I can’t think of off the top of my head.

And what do you need that for?

Well JSON HTTP APIs are awesome in their simplicity, terseness and parsibility. They completely suck at being self-describing, surfable, and evolvable though.

They suck at what?!?

Self-describing

Let’s pretend I’ve got a JSON HTTP API for my fake blog at fakeblog.com/api. Go ahead and add a blog post.

Oh you couldn’t because I didn’t also give you a link to the 22 page documentation wiki?

Hold on to your socks and look at what HTML can do:

<form action="/api/entries" method="POST">
  <input type="text" name="title" />
  <textarea name="post" ></textarea>
</form>

Well that really satisfies the requirement for knowing how to add new blog posts. Sure, there’s a whole lot of crap there that we’d never do in json, but I don’t need to write up a wiki full of docs for you either.

I’m not saying JSON APIs should have forms like HTML.  I am saying they should give you information on how to write to them though.

Surfable

When I write APIs at my company for internal use, people always say “Hey Gregg, what was the URL for that endpoint to do x?”. At first I’d just give them the urls they wanted. Then I’d write docs and pass them urls to the docs. Then I started to get smart and put the URLs for all the endpoints up at the root URL of my API, and I just so happened to put them in a JSON doc as strings.

Well there’s some magic in that that I haven’t been able to fully capitalize on yet. If people write clients to my api that programmatically check this file full of links, then they don’t have to keep a list of all my urls in their code. I can even put links in other API responses that link to endpoints specific to that particular response.

I installed a chrome extension for JSON rendering called JSON view and I could just click around my entire API simply and easily and look for problems on the ‘read’ side of things.

It was brilliant because users could find their way to all my endpoints, including me. I’m far too lazy to go memorizing links and keep copying and pasting them in everywhere. All I needed to do was remember /api and I could surf to any endpoint I wanted to, just like it was a website.

Links as strings are better than not having links, but there are a bunch of valid links (usually relative ones — the ones that don’t start with “http://”) that most people and programs wouldn’t know were links by just reading them. We really just need a JSON format that can denote a link as opposed to a general string.

Evolvable

I haven’t seen this happen yet in the real world, but it sounds awesome. If people aren’t hard-coding your urls in their app everywhere, you can now change your urls whenever you want without breaking them. You’re no longer coupling with all your client apps on the URLs. It can actually go way further than that though. Checkout Jon Moore’s hypermedia API presentation at if you’re interested. It really blew my mind.

If XHTML has all this great stuff, why not just use XHTML?

Forms in XHTML contain a bunch of ‘view’ descriptors that we don’t need (JSON doesn’t care about buttons, or the difference between a text input and a text area!).  XHTML APIs are a pain to parse from javascript and contain a lot of extra cruft you don’t need.  Think of all the reasons that JSON is thoroughly trouncing XML in web-api usage.  People only want the bare minimum featureset in their documents.

So anyway.  That’s what I want.  I have a few ideas that I’m working on, but nothing concrete yet.  I certainly don’t think XHTML features should be verbatim copied into JSON though.  JSON is only for data interchange, and has a simple and terse style that needs to be adhered to.

I’m looking for something that is

• still JSON like XHTML is still XML
• can describe the writable side of the API (like forms do for HTML)
• stays true to the spirit of JSON and doesn’t mindlessly mimic XHTML

I’ll write down my ideas in a later blog post, but I’d love to hear input from other people interested in this (especially those that understand the need to keep it mercilessly simple).  I’ve seen some great ideas like Mike Kelly’s JSON-HAL and Mike Amundsen’s Collection+JSON but those are both a little more specific than I’m talking about.  I’m talking about something less-constrained and more general purpose.

 

This post is also available in Serbo-Croatian thanks to Jovana Milutinovich from Webhostinggeeks.com

Lately I’ve been thinking a lot about the impact of the move to a thick client architecture for web applications, and I’m becoming more and more certain that this means that Rails-style MVC frameworks on the server-side are going to end up being phased out in favour of leaner and meaner frameworks that better address the new needs of thick-client architecture.

There are a few major reasons for this:

The server is no place for view logic

The devices available for running a web app are vastly different from when these web frameworks first sprung up.  The slide towards thicker / richer clients has been proceeding on pace with increases in processing power since the Web 2.0 days.  It’s much simpler to handle views and view logic in only one place, and that place is slowly moving away from the server side.  MVC has always been a strained pattern for a server-side, non-gui application and it is been a confusing and complicated trade-off to have the backend generating front-end logic.    Front-end frameworks like backbone.js, as well as advances in web technologies like HTML5′s history.pushState are now making server-free views a realistic quality of cutting-edge front-ends.   Rendering on the client-side also gives us the opportunity to create responsive designs based on device capability rather than having the server try to somehow figure out what the capabilities are without actually running on that device.

The kinks aren’t all the way out yet, but I do think the trend is clear.

Server-side Templating and .to_json are both underpowered and overpowered for the actual requirements of JSON APIs

There’s no need for templating on the serverside (or view helpers, or any view-related cruft) to generate simple JSON documents, but there are a tonne of problems left unsolved when we fail to see that generating a JSON API is more than just a serialization problem.

How should dates look?  (RFC3339 / ISO8601 of course!).  What should the JSON error document look like when you provide a 400 and want to tell the client why?  How should links to other resources in the API look?  How does a collection look?  How does pagination look?

These aren’t just serialization concerns, and they have nothing to do with templating.

HATEOAS is not just an academic pursuit

A thick client does not want to maintain a vast list of static strings representing all the crazy URLs that it will have to call in a non standard API.  As an API designer, you don’t want them doing this anyway, because hard-coded URLs and URL structures make it a real pain for you to change your API.

The AtomPub protocol, if you ignore its XMLiness,  gets this right — The thick client just knows the root URL (which serves up a simple service document) and is aware of a bunch of relationship types (the ‘rel’ attribute on ‘link’ and ‘a’ tags) that it can follow as necessary.  If a game client needs to access a list of players, it just goes to the root url, and follows the link with the rel property called ‘players’ (and probably remembers that link until the server says it has moved via 301 status code).  JSON has no concept of links or rel, but this is still easy to imagine and implement, and while it’s a teeny bit of extra work up front, the standardization buys you the ability to start writing smarter HTTP/REST clients for your frontend that take care of much more for you automatically, so you can spend time on real business logic and actually do something more productive than fiddle with the javascript version of a routes.rb file.

(A really great API framework might generate some or all of these links on its own and automatically put them in the JSON documents.  It’s pretty easy to imagine pagination links like next/back being generated automatically, or even links amoung resources in some cases, possibly based on some internally declared relationship.)

Rails-style MVC frameworks have a horrible routing solution for RESTFul JSON APIs

In a resource-oriented API, the router need not be concerned with the http methods that are or are not allowed for the resource.  That’s the concern of the resource and the resource alone.  When the router tries to manage that, you get the unnecessary verbosity of a route for every method supported by the resource, and you get the app incorrectly throwing 404s instead of 405s when a method is not supported.  This probably means that ‘controllers’ need to go away in favor of ‘resources’, and routes can be vastly simplified, if not completely derived/inferred by the framework.  Because we keep thinking in this conventional MVC style though, we miss the possibility and potential of vastly more simple applications that actually do a lot more more for us.

The Application Developer shouldn’t have to Deal with these Details

There’s no  reason for us to all separately think about these problems and solve them in a million different ways every time we’re confronted with them.  Aside from the years of wasted time this involves, we’ve also got a bunch of non-standard and sub-standard APIs to interact with, so all the client code needs to be custom as well and nothing is reusable.  This is why being RESTful is not just academic and this is why being concerned with the details is not pedantic.

As I said earlier, AtomPub gets a lot of this right.  A lighter-weight JSON equivalent would be a huge improvement to what people are doing today, because the conventions would mean that the framework can take care of most of these API details that we reimplement ad nauseum — or worse, not at all.  It also means that frontends can start to abstract away more of the HTTP details as well.  This is already starting to happen in the new frontend MVC frameworks but in almost every case, the HTTP end of things still needs to be handled in a custom way for every API endpoint.  There is still way too much work left to the application developer, and we’re silly to continue to do it over and over without coming up with a better abstraction.

CouchApps and WebMachine are just starting to touch on this style of architecture.  Backend-as-a-service platforms like Parse certainly understand how far this simple architecture can go, but ultimately there’s a huge need for a framework that can create more complex RESTful APIs (in a language that’s more general purpose and “blue collar” than Erlang).

Rails-style MVC frameworks are both too much, and not enough at the same time.  It really is time for a new framework to support this new architecture.

Version control branches come with costs. I’m at the point now that I don’t think it’s possible to get away without spending a couple of days every month managing issues with version control and merging if you’re using feature branches liberally as is common with DVCS (like git) users. I’ve spent days myself in the past months, so I felt compelled to write this because those were days I could’ve been actually doing something that adds value instead.

The purpose of branching is to delay integration of some code into the main code line (for various reasons). There’s a heavy cost that comes from this in that the less often you merge, the harder it is to merge. I’m blown away by the effectiveness of automatic conflict resolution in git, but the bottom line is that if two developers are working in the same area of the codebase, semantic conflicts will emerge that need the intervention of a human to resolve. These can be complicated and tedious and involve multiple developers. Yes, branching is easier with DVCS, but this actually exacerbates a fundamental merging problem that DVCS doesn’t solve.

“What if I keep my branches short-lived?”

There are some better defenses for branching, like “What if I keep my branches very short-lived?”, and while it does mitigate most of the problems with merging, it also mitigates any advantage you think you might get from branching. If you’re only going to branch for a day, what’s the point?

“Then how can I pick and choose what gets added/released?”

Merging early and often leads to simpler merges, but what about the case when you get partway through a feature and want to scrap it because you’ve come up with a better way? There are a number of ways to mitigate this problem, but if your need to “unmerge” occurs more often than your need to merge, you’ve probably got larger organizational issues to deal with. If those issues are purely technical, try more prototyping to uncover those issues earlier, and feature-toggles for better control over what gets released. If those organizational issues are non-technical, like frequent changing requirements mid-feature-development then you simply need to get better (and probably smaller) user stories before commencing.

“What if I do daily merges from the mainline into my feature branch?”

You’d think it shouldn’t matter which way you merge if you merge often, but that’s only fine if you’re the only one branching. As soon as someone else on the team comes along and starts up their own feature branch and follows your same best practice, the two of you are diverging from each other and creating merge-debt.

Using a continuous integration server is NOT sufficient for actually practicing continuous integration.

I’ve heard of some teams using a bot to auto-add all branches to the team’s CI server, making sure that each branch gets put through the proper rigor of the CI server. This is actually cargo-cult continuous integration though because those teams are not actually continually integrating at all. Merging early and often is a prerequisite of continuous integration, by definition. Instead, running multiple branches in CI supports maintaining those multiple distinct branches for even longer, so you’re actually using the CI server to undermine the ideals of CI.  Even if you run a special build that tries to auto-merge all branches (yeah right!), but is not the mainline and is not intended for release, you lose the ability to regularly deliver work to some staging environment for manual regression testing.

A small non-distributed team doesn’t have the same problems as Linus

If your team is in the same room and practices collective code ownership (instead of having a central maintainer) you should just do in-person peer code reviews (or better yet, pair programming) instead of relying on pull requests. It’s wasteful to force every aspect of intra-team communication to go through the computers in the room. I would even go so far as to say that if you’re in the same room, you don’t actually need a DVCS.

Most of the reasons behind feature branches seem to be a cure that’s worse than the disease itself. It’s vastly simpler for us to commit to the mainline at all times, early and often, and deal with deactivating or removing particular pieces of code in other ways (like feature-toggles) as those problems arise (which does not in my experience tend to be very often compared to how often I am merging new stuff in that stays in).

I don’t mean to be negative about DVCS either. I actually use git exclusively now, I think github has been a huge boon to open source, and I love anything that makes Linus more effective at maintaining my favourite OS kernel. It just really seems like feature-branches have never solved any real problem that I have, while they have caused a lot of pain.