I’ve seen these anti-patterns in a lot of web APIs, so I thought I’d detail why they’re bad and what to do instead.

Using a 200 status code for every response:

This anti-pattern is common for SOAP-like RPC APIs. Using a 200 status code for all your responses is silly because you end up having to reinvent an error-indication in your response body instead, and nobody in the world knows your new standard (or wants to have to learn it).

It’s also extremely confusing in the cases that an error has occurred because a status code of 200 means everything is A-OK. There are already a bunch of error codes in the 400 and 500 ranges that are fully capable of indicating all sorts of common problems. It’s still great to put an error message in the response body, but there should be a 400 or 500-level status code so people know that an error even occurred and don’t go ahead and try to interpret the response body as normal or otherwise assume a success.

Using a 500 status code for all errors:

This anti-pattern is common because 500 is what most frameworks will do when a web application throws an uncaught exception, and so it becomes convenient to just throw exceptions and not worry about the repercussions. This is mildly better than using a 200 status code for literally everything, because at least it shows an error occurred but it still causes a lot of confusion for client developers.

When you 500 for an error (even if your web framework did it for you), you’re signalling that the problem with the request was the fault of your application (the server) and it’s possible that retrying the request will be successful. Because 500 errors indicate an internal server error, they shouldn’t actually ever happen though1 — In general, you’ve got a bug in your application if they do, and the web server will tell everyone about it. The right thing to do, if there’s something wrong with the request is to respond with the appropriate 400-level status code.

Using a 500 error when a 400-level status code is more appropriate is obviously really confusing to the person making the requests because it doesn’t even let them know that their request needs to be fixed. Even if that person is also you, it forces you to start digging into your back-end for what could possibly just be a bad client request. If you don’t know which 400-level code to use, at the very least use 400 (Bad Request) itself. This will at least give your client developer an indication if the blame is on the server or the client and cut your debugging in half. A few words about the error in the response body can go a long way as well.

Logging 400-level codes at ‘error’ level:

Logging is pretty important to measuring the quality of implementation of an API, and learning about specific defects. The ‘error’ level in your logs should be a way to indicate “things that should not have happened that are within this application’s control”, so that you actually have a hope of attempting a 0-error policy. 400 level codes are errors of course, but they’re client errors, and not your server application’s error, so they should not be logged as errors along with your actual 500-level errors (Of course you might still log them as “info” level to keep track of what kinds of crazy things your clients are doing).

Why do all these details matter?

When you have an interface between two applications (the client and the server in this case) like HTTP, it REALLY simplifies things a lot if they communicate whether messages are acceptable or not (and why). Even if you control both ends of the interaction, you can save yourself a lot of server-side debugging by having the server indicate what exactly went wrong with a given request, instead of repeatedly reading stack traces every time some JSON you POSTed is missing a property. Compared to how little extra work there is involved in communicating the problem correctly, it’s pretty short-sighted to just let all exceptions bubble-up, or catch all exceptions silently (and 200).

Keep in mind too that if you’re now convinced that communicating errors is important, then HTTP already has a system for just that: HTTP status codes (plus whatever specifics you want to additionally communicate in the response body). Status codes are part of the greater HTTP standard that a lot of developers, libraries, proxies, and caches already understand. Re-inventing your own error mechanism is generally a waste of everyone’s time (with you at the top of the list).

Getting this right is a real time-saver over the long-haul even if you’re the only consumer of the API writing both client-side and server-side. The time-savings only multiplies as you add more clients.


  1. Sometimes 500 is appropriate of course: If your application requires a database and its lost connectivity to it, then a 500 makes total sense. It signals that there’s a problem in your application and the client can possibly try the request again later.

If you consider the number of packages on pypi, npm, and rubygems, and the release dates of python (1991), ruby (1995) and node.js (2009), it looks a lot like those communities are getting new packages at the following rates, per year:

python: 29,720 packages / 22 years = 1351 packages per year

ruby: 54,385 packages / 18 years = 3022 packages per year

node.js 26,966 packages / 4 years = 6742 packages per year

It’s important to note that I’m only showing the other languages as a measuring stick. There are probably a lot of reasons for the disparity here (including my imprecise math), and I’m not trying to say anything like “node.js > ruby > python” but obviously new node.js packages are being published to npm at a staggering rate. I just checked npm (on a Sunday night) and 20 new versions of packages have been published to npm in the last hour alone.

This is a pretty amazing feat for node.js. How is this possible?

Batteries NOT included

If you’ve needed an HTTP client in Python, you’ve probably asked yourself “should I use urllib, urllib2, or httplib for this?” like so many people before you. Well the answer is probably that you should use requests. It’s just a really simple, intuitive HTTP client library that wraps up a lot of weirdness and bugs from the standard libraries, but it’s not a standard library like the others.

The “Batteries included” philosophy of Python was definitely the right approach during the mid 90’s and one of the reasons that I loved Python so much; this was a time before modern package management, and before it was easy to find and install community-created libraries. Nowadays though I think it’s counter-productive. Developers in the community rarely want to bother trying to compete with the standard library, so people are less likely to try to write libraries that improve upon it.

Developers are less likely to use non-standard libraries in their projects too. I’ve heard many instance of people suffering through using an inferior standard library just to have “no dependencies”. But in this day and age of cheap massive storage and modern package management, there are very few reasons for a policy of “no dependencies”.

Conversely, the node.js core developers seem to actually want to minimize the standard library as much as possible. They have repeatedly removed features from the standard library to free them up to be implemented by the community instead. This allows for the most variety and lets the best implementation win (but only until someone makes a better one, of course!).

Imagine how liberating this is for standard library maintainers too. The node.js standard library is comparatively tiny, freeing the core developers to just deal with the core necessities.

The Tiny Module Aesthetic

Just like the 140 character limit on twitter makes people “blog” more, the node.js community has a culture of publishing tiny modules that makes people more comfortable with publishing smaller packages, and so it happens vastly more often.

While I don’t really consider myself “a node.js developer”, I’ve found myself publishing over a dozen packages in the last year myself, just because I’m working on a node.js project at work, and I publish any reusable package from that work that I create. My project ends up factored better, and I end up with a bunch of building blocks I can re-use elsewhere (and then I do!). Sometimes random people from the interwebs even fix my bugs before I notice them!

This approach is not unusual at all either, at least in the node.js community. The community is lead by developers that have each published literally hundreds of packages. I don’t even know how this is possible.

Of course there’s absolutely nothing to stop other languages from following suit, so it seems to me like community culture is the deciding factor.

Massive monolithic frameworks like Ruby on Rails tend to not take hold in the node.js community mindshare, I think due in part to this “tiny module aesthetic”, and it’s all the better for it. It seems to me that monolithic frameworks like Rails don’t have a clear notion of what does not belong in Rails, and so they tend to expand indefinitely to include everything an application could need. This leads to people trying to assert that Rails is the best solution for all problems, and having a cooling effect on innovation within the web application space in the Ruby community.

The tiny module aesthetic also leads to an ecosystem of extreme re-use. When a package doesn’t do more than is absolutely necessary, it’s very easy to understand and to integrate into other applications. Conversely, while massive monolithic frameworks tend to appear in a lot of end-user applications, they rarely never end up being dependencies in other packages themselves. It seems unfortunate to me that so much great software should be so difficult to re-use.

git and github.com monoculture

To be honest, few things are more boring to me in software development than version control, and I find the complexity of git to be a bit silly and mostly unnecessary for me, but I’ve never heard of a node.js developer that doesn’t use it, and there’s huge value in that kind of monoculture. It means that they all speak with a common language and there’s no impedance when you want to contribute to someone else’s projects.

Github.com has a similar effect in lowering the barriers for cross-pollination between developers. I’ve only very rarely seen a node.js project that wasn’t on github. This means I immediately know exactly where to find source code if I want to contribute.

The pluses and minuses of a github monoculture go far beyond this, but there’s probably enough fodder there for an entirely different post.

Permissive Licensing

Node.js packages tend to use extremely permissive licensing like MIT and BSD licensing. In fact, the default license for a package created by npm init is BSD. I think this is another sign of the times.

Very few people care anymore if someone else forks and doesn’t contribute back. There’s very little to gain in not contributing back anyway because of the cost of maintaining your own fork.

This is important because no one wants to have to deal with dependencies that make them think about legal repercussions, or give them extra responsibilities. Licenses like the GPL end up having a cooling effect on software-reuse for this exact reason.


EDIT Jun 3rd, 2014: Comments below are basically copy/pasted from my old blog format for posterity.


30 thoughts on “The node.js Community is Quietly Changing the Face of Open Source

  1. max ogden said on :

    Nice writeup! I do think that the term ‘node community’ is a little bit too general since some of the most popular node modules are in fact bigger than they should be.

    For example, the 4 most popular modules from https://npmjs.org/: underscore, async, request and express could all be split up into smaller pieces:

    Underscore isn’t terribly large but it has things tucked in it like a templating function that don’t belong.

    Async is thematically consistent, offering many variations on a theme, but from the empirical usage patterns that I’ve seen most people use a small percentage of the offered functionality.

    Request actually went through the modularization process over the last month or two and went from having 1 or 2 dependencies to having 11.

    Express is perhaps the greediest module on NPM (IMHO) in that it invented its own naive APIs early on which means that now you can only use modules designed for express with express since the extensibility is stuck in a relatively pre-cambrian era. Classic framework lock-in, albeit to a lesser extent than other, larger web frameworks. I’ve thought a lot about why this happened and believe that if you try to make a land-grab early on and abstract too much too early then you get stuck with your abstractions due to backwards compatibility. Good software grows slowly.

  2. Yeah I totally agree, and you’ve probably explained it even better than I did.

    Express is actually a bit of a conundrum for me for a few reasons, especially since in trying to create a mini API framework myself ( http://percolatorjs.com ), I’ve felt forced into duplicating some of the things I dislike about the modularity (or lack of modularity) in express, to the point that I eventually gave up and added ‘connect’ middleware support. There are a few competing pressures that make web frameworks like express (and rails for that matter) popular:

    (1) Some people just want to “get shit done” ™ and don’t want to have to search for multiple pieces of a puzzle to hobble together a solution. I think there’s probably a gap where finding the best modules for a given problem is still a too hard. Filling that gap (somehow) will promote collaboration even more. The current npm search (on its own) doesn’t solve the problem IMO.

    (2) HTTP is actually really complicated and very hard to abstract effectively. For example, how many frameworks 404 when you try an invalid method on an existing resource instead of the proper 405? This is mostly because of the way routing is abstracted; These frameworks use the http method as the first ‘key’ in the routing table, when really the url makes more sense as a key and the method should be secondary. There are a number of cross-cutting aspects like this including logging, error-handling, and authentication where some sort of centralized pre-processing makes a lot of sense. Not all “aspects” should be handled with serialized pre-processing middleware though, and unfortunately that’s what we’re seeing a proliferation of. I’m disappointed that I can’t build a mini-framework without integrating connect, or rewriting/repackaging all those otherwise great connect middleware modules out there, but I don’t know a better way. To be honest, I don’t even know at this point what the better abstraction would be. Doing everything in pre-processing middleware and monkey-patching the request/response objects doesn’t seem like the answer though.

  3. Good stuff. I haven’t felt this excited about a platform community in a long time.

  4. Good read: spot on.

    I would add a couple additional factors to that trend:

    - Proper package management: `npm` was and still is what got me hooked up. Package management is often an afterthought that is rarely properly addressed.

    - Flat community: which is partly illustrated with the extreme culture of re-use you mentioned. I feel there is much lower friction in creating and contributing to modules in the node.js community, with no gate keeper in sight.

  5. For a similar system: Perl’s CPAN (the package repository) has been online since 1995 and presently has 120,447 modules for a yearly rate of 6691 modules for 18 years now.

    It’s been providing the same functionality you mention: an encouraged license (“the same terms as Perl”), a monoculture of software (CPAN for package distribution/documentation), and an emphasis on tiny modules. Little packages are encouraged and really the way to go when you’re building Perl modules.

    That said, it’s great to see those practices being adopted successfully by Node and seeing the success of the groups using it. We live in very exciting times for open source. :)

  6. The “The Tiny Module Aesthetic” reminds me that, the emerging of the “Single-purpose app” in recent years, and the design philology of the Unix utilities. And I think those are all about “simplicity”, which I think and care about the more and more in recent years, but I always have to remind myself about when writing code for LIVEditor – my code editor with “Firebug” embedded.

  7. Mansur said on :

    There are some frameworks in Node.JS which are just copied, not unique, like for example https://github.com/biggora/caminte – is just a copy of Juggling DB https://github.com/1602/jugglingdb.
    I would strongly recommend to use genuine frameworks which have been tested by the community and better supported.

  8. Why don’t you consider Perl? 120,460 Perl modules in 15 years, that makes it 8030 modules per year and it’s increasing exponentially now.

  9. PeterisP said on :

    “Batteries not included” approach puts a significant cost on application developers.
    If I need to do a simple thing X, I’d rather have a single good library for X rather than twenty libraries with varying focus and quality. Even if one of those twenty libraries would be far better for my needs, the task of properly comparing and choosing the lib is often higher than doing X with a random library.

    It also reduces maintainability – if one project uses lib1 for X, and another project uses lib2 with a different API for the exact same thing, then it makes stuff harder to maintain.

  10. @ Chankey Pathak, Mark Smith: For the record, I didn’t include perl because I haven’t got any perl experience to draw from, while I’ve spent years writing reams of code in ruby, python and node.js. I can honestly say though that CPAN is nothing short of legendary even outside the perl community, and npmjs.org would do well to follow its lead. It would be interesting to know too how CPAN maintains that rate of growth, and if it’s due to the same things I’ve mentioned.

    I should also add that node.js isn’t even the language/platform that’s nearest and dearest to my heart. I’d really just love my other favorite languages’ ecosystems to pick up some of these qualities.

  11. Su-Shee said on :

    From a CPAN’s lover’s perspective NPM already is _very_ well done and pretty smooth to use.

    From a Perl lover’s perspective, JavaScript doesn’t have a choice anyways, because you really NEED a lot of modules to work around well let’s politely say “language quirks”. ;)

    You simply need things like underscore or node, because it gives you things which aren’t builtin in the language. (*cough* npm install printf *cough* ;)

    Similarily, Perl has gotten its Lisp-style MOP OO module Moose, because there’s no such thing as a “class” in Perl.

    Our equivalent of underscore-alike modules are for example List::Utils, Lists::MoreUtils etc. etc. – and you can’t possibly shove ALL those modules into what we call “core” – set aside that half of the people want Foo and the other half Bar and then there’s our “problem” we’re really proud of: Backwards compatibility is _amazing_ in Perl and there’s always someone who runs a 12 year old Perl and still needs his Really::Ancient::Module::BlahBlah. ;)

    So far I encountered exactly ONE npm module which I couldn’t install blindly with npm install -g – and that was due to an architecture check in its requirements (which isn’t necessary, module very high level) and my Linux flavor returns a different architecture string – so one lesson might be to encourage a best practise “don’t overdo requirements for your module, it just complicates things”

    In Perl, it’s common that you simply ask “the community” (IRC for example) “what do I want for XY?” and get a variety of 2-5 modules and a short explanation why you want A and not B. Imagine for example you have more than one XML/HTML parser module (which we do) – do you want XPath support or do you prefer DOM-style methods? Do you need a strict parser or do you want to work on dirty HTML?

    Most module authors mention these kinds of use cases and “flavors” to do things and refer graciously to the alternatives. Plus, there’s the entire commenting and rating system for modules and yes, you of course look if Perl gurus Tom, Dick and Harry Superperl ranks module Foo highly or not – it’s not that we don’t have our own little rockstars and divas. ;)

    Due to variety and so many choices, there’s also bad choices and we put A LOT of work into discouraging certain modules these days which are simply not really appropriate to use in 2013 or some well known badly written ones. But you can’t just throw them out, too – because Perl is installed everywhere – what is the browser for JavaScript is pretty much any Linux/Unix for Perl. And all the shitty JavaScript quirks you’ve gotten from browser vendors, we get from distributors which come with a Perl from 1999 in 2013.

    Sooner rather than later (there already are a couple) there will be a class of modules on NPM which are dual use – browser and Node-based server-side and you will have to keep that straight and compatible and that will be a lot of work. Also, what if people start writing JavaScript modules which are able to run in browser, on Node AND on e.g. spidermonkey? (what about Rhino/Nashorn?) I’d encourage that a lot, not everything needs Node and NPM is the de facto thing to put a module up on by now.

    Another advantage of CPAN is: It’s _extremely_ centralized. You look for a module on CPAN. Period. There’s no “download from here” or “fetch from there” – it’s on CPAN or it doesn’t exist. As CPAN is tested and tested and retested by volunteers, using modules from CPAN pretty much ensures been tested on a dozen platforms. If you use the plain “cpan Foo::Bar” installer, you can always watch all tests live while running the installation of a module.

    And Perl modules have tests LIKE THERE’S NO TOMORROW – as Perl’s testing flavor TAP is extremely easy and simply and cheaply to do, modules actually _do_ have a LOT of tests.

    Last but not least: On average, the commonly used Perl modules are _extremely_ well documented and the docs can be read on commandline and via web. The documentation style is probably rather old-school from a Node lover’s perspective – it’s more on the dry, informative side and less about being cute and funny – but some docs literally go into the amount of a book written.

    Also, no matter how big a project – I simply expect it to be installable via CPAN. There’s a couple of Perl things which are more applications but “modules” and are not installable via CPAN, but that’s pretty rare. Entire frameworks are installed via CPAN – think like the difference of how you install Derby.js and Meteor.js.

    And: I run deliberately large dependency-chains for the fun of it – CPAN has a flag for “resolve all dependencies yourself and say auto-yes to everything you’re asked during installation” and I manage a couple of hundred modules in ONE session WITHOUT any error while running ALL tests during installation – possibly on half a dozen Perl versions (Made with perlbrew which is our nvm/rvm/virtualenv).

    As the Perl mantra “there’s more than one way to do it” is done by the JavaScript community on the same level of excess (a dozen web frameworks anyone? ;) you’ve made your choice already anyways to go down the road of extreme flexibility and variety. It already happened, no going back here.

    The Perl community for example is very forceful to really push people into using modules from CPAN exactly BECAUSE they’re tested and have solved hundreds of problems for you – you don’t split a CSV file by hand, you use Text::CSV which is proven to work for a decade. You won’t do better with your meager little split. :)

    You don’t parse HTML with regex, you use a module. You don’t validate email addresses by hand, you use a module. We have modules like Regex::Common for example which contain a range of common usecases people usally throw (bad) regex on or modules which pretty much read any stupid format known to mankind – it’s all put into modules and more modules.

    Luckily, Perl also has most of the necessary modules I usally call “boring shit” modules – Excel reading and writing anyone? Those are the true heroes in my opinion – people who sit down and really write this kind of ungrateful, boring stuff you suddenly need there’s *gasp* lurking an “enterprise” around the corner – not to mention that nobody in their right mind is going to implement SSL bindings three times. (One lesson for JavaScript/Node might be to maybe not add another web framework but deliberately go for the boring stuff sooner rather than later and get it over with..)

    For my part, I can’t rave enough about CPAN and some of the people poured their hearts and time and knowledge into some modules. Billion euro/dollar businesses run on it – some for over a decade straight.

    So, hack, upload and enjoy. :)

    Yours truely – a Perl lover. :)

    CPAN’s main website: http://metacpan.org

  12. subdigit said on :

    Maven is also something that’s begun to pick up pace in the Java community. Stats seem to be here: http://search.maven.org/#stats

    But not sure about exact growth rate. A pretty picture is available here: http://mvnrepository.com/

    While having a small standard library is great for essentially encouraging reinventing the wheel to make it better, it can at times be a detriment for “just getting things done”. It’s called a “standard” for a reason, which is that you can rely on it to be present and available for use, and more importantly, for understanding by the developers that come in and out of a project.

    That being said, popular extensions will solidify and essentially become a 3rd party standard for use and becomes indispensable and tied into the core project, whether it’s just the common-knowledge standard du jour, or it actually gets merged in.

    It certainly is a more interesting way to get a diversity of development of what people consider to be core though. For that, I find that model interesting to watch and see how it grows and if anything I favor “wins” :).

  13. Su-Shee: thanks for the education! I agree, CPAN sounds great. Since we’re getting along so well, I won’t say anything about the language itself. :)

    A few minor clarifications on javascript in 2013: Don’t `npm install printf`, use util.format: http://nodejs.org/api/util.html#util_util_format_format . Underscore is much less necessary nowadays (especially in node.js) as a result of ecmascript 1.6 improvements (See “iteration methods” at https://developer.mozilla.org/en-US/docs/JavaScript/Reference/Global_Objects/Array ). Also using jslint will keep you 99% quirk-free (I have it integrated into my builds and vim).

  14. There’s no need to slander the GPL in this post, it’s just as permissive and would allow the same type of collaboration. The trouble is that most developers also want to use libraries as part of proprietary code bases. I don’t understand why they don’t just dual-license; GPL for non-profits and devs, MIT/BSD for commercial/proprietary software.

    There really should be a bigger push to license internal code under the GPL. 99% of the time it doesn’t offer a competitive edge. It’s the data and marketing and network effects that have more to do with it than anything.

  15. Andrew de Andrade said on :

    Great post. I’d also add to that list the benefits of local by default packages that are in your project folder. The approach of tools like easy_install, pip, bundler, etc is to hide all your depemdencies in some dot-prefixed folder outside your project directory. This promotes an out of sight, out of mind mentality. The npm approach on the other hand puts them in your project and makes it much more likely that you are going to explore the source of your dependencies, more likely that you bug fix those dependencies instead of work around bugs and most importantly you’ll treat that folder like part of your own project. This last point means that you are likely to develop one of your own generic modules to the point where it’s good enough to submit back to npm as a public module.

  16. Su-Shee said on :

    Oh, go ahead and say something about the language and I will probably tell you the exact same thing back about JavaScript – or something quite similar. If anything, both have A LOT of history and baggage – but they both also have A LOT of flexibility, otherwise the range of modules existing as of today wouldn’t even be possible and people wouldn’t deliver at that speed. And let’s face it: We can happily unite (Come along, you too Java, don’t be shy…) in Python’s and Ruby’s dislike for {} and ; – and we use a lot of them. ;)

    Sadly, format.util supported too few options when I looked last. (In the end, I did it in DB anyways..) – but it’ll evolve, I’m sure.

    Underscore will probably move to other list-supporting functions over the years I would expect, the map-grep-filter-foreach range is just the basics anyways. (At least, Perl didn’t need map-grep-foreach-shift-unshift-push-pop-splice as a module, that really is core-core…)

    And before someone is tempted to bitch about Java-and-Maven: _THAT_ community delivered literally EVERYTHING I can think up of “boring shit” and “enterprisey stuff” you can imagine. And if you ever looked for “good PDF manipulation” for example: Java did deliver. Kudos for that.

    If anything the lesson simply is: Yes, you want TONS of modules, yes, the environment IS important, no, it doesn’t matter if you’re a less-than-perfect language, as long as you support a certain flexibility, people will work around it.

    And style evolves a lot – nobody writes JavaScript like in 2000 anymore and you get discouraged a lot to do so – same goes for Perl. But if you already had a web business in 2000, you probably just can’t throw out your entire carefully crafted JavaScript or your backend of the day (in 2000) and that is something EVERYBODY faces. Perl did and we paid dearly, Java does, Ruby will, Python will do too.

    JavaScript showed – thanks to Douglas Crockford and jQuery among others and yes, thanks to Node.js, too – how to evolve nicely into a new style – but also Node.js setups will at some point be really huge and not that easily updated anymore and then you face the very same thing everybody else does.

    And god do I wish this petty bickering among languages would stop – we owe each other so many things – all of us – and all of us owe Lisp and Smalltalk and C.

    Let me all remind you that there was a time when “real programmers” laughed about using a “scripting language” on this cheap little Linux hack with this meh little HTML thingie to put up a business on and now look where we all are today…

  17. Rudolph: “Slander” is a legal term and a pretty ridiculous one for what I said. More legalese isn’t going to help the GPL’s case. You already agree that the GPL is more restrictive for people mixing software with other proprietary software, so how can it *possibly* be slanderous for me to say it’s not as permissive?

  18. Hisham said on :

    Well, counting the age of the Python and Ruby repos by the age of the language (and not of the package management projects, such as pypi and rubygems) while considering node.js by the age of the project and not the language (JavaScript) skews the numbers quite a bit!

    Let’s compare by language age:

    python: 29,720 packages / 22 years (1991) = 1351 packages per year

    ruby: 54,385 packages / 18 years (1995) = 3022 packages per year

    node.js 26,966 packages / 18 years (1995) = 1498 packages per year

    And by package manager age:

    python: 29,720 packages / 11 years (2002?) = 2701 packages per year

    ruby: 54,385 packages / 10 years (2003) = 5438 packages per year

    node.js 26,966 packages / 4 years (2009) = 6742 packages per year

    Except that, of course, these numbers are meaningless, because each language/tools has its own submission evaluation process (including “no evaluation”), and also different granularities in how to count “packages” (by module? by a set of modules that compose a project?).

    Also, language-specific package managers are more or less a new thing — counting for example the number of packages submitted last year would be a more meaningful metric (apart from the problems I raised above.)

    Still, it’s nice to see that language-specifc package managers are thriving, and I wish all the best to all of them. :)

  19. TJ said on :

    The reason Express is the size it is, is because all these things are what we would be duplicating each time anyway. Yes a lot of it could/should/will live in npm, but an aggregate is not harmful. Look at npm-www for example, this is what happens when you re-create things every single time you write an app, it becomes are large unwieldy mess of stitching together random things and re-inventing within the app itself instead of an aggregate “framework”. Most middleware should absolutely be abstracted to a lower level and then “wrapped” as middleware, there’s no question about that, but should they cease to exist? absolute not.

  20. TJ: Yeah, I actually agree that the core http module isn’t a useful enough abstraction to be used for many (most?) real world web applications. It also doesn’t offer a reasonable way to add functionality other than wrapping it or monkey-patching the request/response objects. Cross-cutting concerns are the obvious problem (security, logging, routing, etc), but also: what if you want to just add functionality that’s more easily accessible than constantly requiring the same 20 modules that are all very HTTP relevant? And I agree that larger aggregates are useful and valid (and I bet you’d also agree conversely that putting an orm in a web server abstraction is a superfluous aggregation, and so there are limits to what aggregates make sense).

    I don’t actually have a better solution than what express does. I hope that was obvious from my comments. I certainly tried, but ended up coming back to connect. At the same time, I find it unfortunate that connect middleware have non-obvious interdependencies and are difficult to keep clash-free (basically all the problems of monkey-patching). But like I say, I haven’t got a better proposal.

    I personally don’t think node.js has the HTTP abstraction problem solved very well (yet).

    > Most middleware should absolutely be abstracted to a lower level and then “wrapped” as middleware, there’s no question about that

    Not a bad start!

  21. @Su-Shee: If there would be an upvote button on comments here I would’ve hit it so hard that it would break my mouse! Well written, +1 to you.

  22. Some times I get confused when people try to convince other’s programmers that easily re-usuable modules are awesome and you should try them. Then I remember that not every was into perl 10 years a go… Even if you’re not a fan of the perl syntax (beauty), CPAN is the shit and so is NPM.

  23. Pingback: This Week's Links: _why Edition - NYTimes.com

  24. Adam said on :

    I’m sure this was already said.. but you can’t compare node.js to php/ruby. I am a node developer.. but node is a new FRAMEWORK. It would be more accurate to compare JAVASCRIPT to php/ruby.

    Just FYI

  25. :

    Adam: The comparisons I make are comparisons in the customs of the communities surrounding those technologies and not the technologies themselves. I think at that level, it’s apples-to-apples.

  26. Juzer Ali said on :

    @PeterisP:
    > “Batteries not included” approach puts a significant cost on application developers.
    I agree, it sure wastes a hell lot of time.

    @TJ: Is express code too succinct for what it does? For example, its very difficult to figure out what is happening inside a router https://github.com/visionmedia/express/blob/master/lib/router/index.js#L46

  27. Pingback: To node or not to node – overview of Node.js | Developing Scalable Applications 

  28. Simon Boudrias said on :

    @Adam I have to disagree here. JavaScript is only a language (it is syntax), PHP/Ruby are more than syntax, it is syntax + all about the runtime and functions hooks to the environment. Node.js is JavaScript + runtime/hooks stuff. That’s why Node.js is comparable to PHP and Ruby, JavaScript isn’t.

  29. Pingback: ~ The Truth about JavaScript. « Clint Nash 

If you’ve developed SaaS in a large organization (or even a small organization that acts like one), you’ve probably seen some of the symptoms of fear of releasing software:

  • Off-hours deployments and weekend deployments
  • Infrequent releases
  • extended code-freeze and testing cycles
  • Reluctance to refactor
  • Feature branches for the purpose of ‘cherry-picking’
  • release managers
  • many non-customer-facing environments before production.

These symptoms usually happen only in larger/older organizations, because of two factors:

  • They take a lot more personnel / time to implement than smaller organizations have.
  • They are usually processes and attitudes that are collected as a reaction to problems that occur in the course of a company’s lifetime.

“But you can’t be too careful!”

One problem is that you CAN indeed be too careful. It’s absolutely imperative that the costs be considered as well, if you want software development to still go reasonably fast. Otherwise these manual preventative processes will slowly creep in over time and your team will be releasing at a snail’s pace.

The other problem is that these policies alone are in some ways not careful enough. Where are the techniques for mitigating defects that inevitably still slip through? Where are the techniques to detect them, limit the customers exposed, and limit the time that customers are exposed?

All of these symptoms have a few things in common: They tend toward heavy manual and preventative process added generously over time, usually with little thought to the cost/benefit ratio of such process. It’s all too common for every production issue to result in a new process to prevent it, without regard to how infrequently that problem might occur, how damaging the result is, or how easily it is rectified.

So how do we keep bugs from getting to our customers?!?

Non-preventative measures (I call them “Reactive Measures“) do indeed exist, and can help you safely maintain a fast and steady flow of new features to customers.

  • Feature-Flags allow you to commit incomplete work to master and push it to production safely. Customer-specific feature flags allow you to test in production or enroll customers in a beta program. With a good feature-flag system, it’s even possible to release a feature to only a certain percentage of customers, and to watch your metrics to see the effects.
  • Easy/fast (read “Automated”) deployment. If deployment takes a long time, then you can’t afford to deploy often. Also, if you can’t deploy quickly, you can’t redeploy quickly (in the event of a problem).
  • Easy-undo. Rolling back from a defective release to a known-good release is the simplest and fastest way to recover from defects, if the process is both quick and easy. Defects will happen no matter how much prevention you use: Easy-undo can mean the difference between light customer impact and heavy customer impact.
  • Smaller releases. Smaller releases have less code, and therefore generally less bugs. Because they don’t contain many changes, they’re easier to test and a rollback is less-likely to negatively impact a customer. Also when a production defect appears in the latest release you have much less code to sift through to determine the cause.
  • Practice Continuous Integration (which necessarily involves always committing to master, and only incidentally has to do with using a CI server) . Feature branches, infrequent commits, and committing non-production-ready work are all practices that need to be avoided, because they are all time-consuming delaying mechanisms.
  • “Done” == “In Production”. If the team moves on to new features before the last one is in production, then that last feature is obviously unnecessarily being held back from release. They will be faster if they deliver to production and ensure it functions there properly before moving on to new features because it avoids unnecessary and time-consuming multi-tasking. This goes hand-in-hand with smaller, more frequent releases.
  • Comprehensive Automated testing. When you deploy more often, you need to test more often. The cheapest way to do this is with comprehensive automated tests. While they’re a heavy-upfront cost, if you run them as often as possible, they’ll pay for themselves very shortly.
  • Continuous Production testing. Create automated black-box integration tests for some of your most important features and figure out a way to run them repeatedly and continuously against your production deployment so that you know instantly when those features are broken. There’s no need to wait for customer feedback to learn about a faulty release.
  • Searchable, aggregated logging with proper log-levels. Making your logs easily searchable in one place makes them something that developers will actually use for defect detection and for debugging. They can go a long way to making your application more transparent.
  • Runtime monitoring, runtime metrics and information radiators (most importantly alerts). Instrument the things you care about most, and have alerts sent out when the measurements are abnormal.

Note: Of course some preventative processes are necessary, but when they are, automation is almost always preferable to time-consuming manual human processes. Either way, you should consider the cost of the process and determine if it the fall-out from the type of defect it prevents is actually worth that cost.

In the end, a fear of releasing is a fear of change, and a fear of change is a fear of improvement. This is a crucial lesson that Big Company software teams can learn from the small, scrappy start-ups that are quickly trying to disrupt them. The more often you release, the more often you can test hypotheses and get customer feedback, and the more often you can course-correct. Companies that don’t realize this are susceptible to being leap-frogged by competitors that do.

Version control branches come with costs. I’m at the point now that I don’t think it’s possible to get away without spending a couple of days every month managing issues with version control and merging if you’re using feature branches liberally as is common with DVCS (like git) users. I’ve spent days myself in the past months, so I felt compelled to write this because those were days I could’ve been actually doing something that adds value instead.

The purpose of branching is to delay integration of some code into the main code line (for various reasons). There’s a heavy cost that comes from this in that the less often you merge, the harder it is to merge. I’m blown away by the effectiveness of automatic conflict resolution in git, but the bottom line is that if two developers are working in the same area of the codebase, semantic conflicts will emerge that need the intervention of a human to resolve. These can be complicated and tedious and involve multiple developers. Yes, branching is easier with DVCS, but this actually exacerbates a fundamental merging problem that DVCS doesn’t solve.

“What if I keep my branches short-lived?”

There are some better defenses for branching, like “What if I keep my branches very short-lived?”, and while it does mitigate most of the problems with merging, it also mitigates any advantage you think you might get from branching. If you’re only going to branch for a day, what’s the point?

“Then how can I pick and choose what gets added/released?”

Merging early and often leads to simpler merges, but what about the case when you get partway through a feature and want to scrap it because you’ve come up with a better way? There are a number of ways to mitigate this problem, but if your need to “unmerge” occurs more often than your need to merge, you’ve probably got larger organizational issues to deal with. If those issues are purely technical, try more prototyping to uncover those issues earlier, and feature-toggles for better control over what gets released. If those organizational issues are non-technical, like frequent changing requirements mid-feature-development then you simply need to get better (and probably smaller) user stories before commencing.

“What if I do daily merges from the mainline into my feature branch?”

You’d think it shouldn’t matter which way you merge if you merge often, but that’s only fine if you’re the only one branching. As soon as someone else on the team comes along and starts up their own feature branch and follows your same best practice, the two of you are diverging from each other and creating merge-debt.

Using a continuous integration server is NOT sufficient for actually practicing continuous integration.

I’ve heard of some teams using a bot to auto-add all branches to the team’s CI server, making sure that each branch gets put through the proper rigor of the CI server. This is actually cargo-cult continuous integration though because those teams are not actually continually integrating at all. Merging early and often is a prerequisite of continuous integration, by definition. Instead, running multiple branches in CI supports maintaining those multiple distinct branches for even longer, so you’re actually using the CI server to undermine the ideals of CI. Even if you run a special build that tries to auto-merge all branches (yeah right!), but is not the mainline and is not intended for release, you lose the ability to regularly deliver work to some staging environment for manual regression testing.

A small non-distributed team doesn’t have the same problems as Linus

If your team is in the same room and practices collective code ownership (instead of having a central maintainer) you should just do in-person peer code reviews (or better yet, pair programming) instead of relying on pull requests. It’s wasteful to force every aspect of intra-team communication to go through the computers in the room. I would even go so far as to say that if you’re in the same room, you don’t actually need a DVCS.

Most of the reasons behind feature branches seem to be a cure that’s worse than the disease itself. It’s vastly simpler for us to commit to the mainline at all times, early and often, and deal with deactivating or removing particular pieces of code in other ways (like feature-toggles) as those problems arise (which does not in my experience tend to be very often compared to how often I am merging new stuff in that stays in).

I don’t mean to be negative about DVCS either. I actually use git exclusively now, I think github has been a huge boon to open source, and I love anything that makes Linus more effective at maintaining my favourite OS kernel. It just really seems like feature-branches have never solved any real problem that I have, while they have caused a lot of pain.

Let’s just forget about story points, pigs, chickens, daily stand-ups, burndown charts, sprints and scrummasters for a second. In my experience all of that weird jargon is really hurting Agile’s adoption among the no-nonsense type of developer that doesn’t have time for fluff and wants to know the real benefits of something before investing time in learning about it (instead of a million other things they could be learning).

I’ve also seen that when anyone happens to cite a failed experience with Agile, the explanation from Agilists is invariably the cryptic “you weren’t doing it right”. This is a bit of an irritating response to anyone that’s spent a significant amount of time following the ceremonies and speaking the lingo though. Instead I’d like to submit that you have a much better chance of succeeding with Agile if you know the real reasons behind the madness.

There are a number of Agile values I’ve noticed in my years of practicing scrum and XP that I think stand on their own even if you’re not doing Agile and just want to improve your development process in some small way. I’m going to list them (and try to stay jargon-free) because I imagine that they would be useful individually, but they do have a sort of self-reinforcing quality to them where their effects will somehow be amplified if you use them together. You should also feel free to break the rules where you feel they need to be broken, but I’ve very rarely found that to be necessary. Usually when I’ve broken these rules, I’ve regretted it later, because what I thought was an exception didn’t really end up being an exception.

Create a team, not just a group of individuals.

A project will move faster when it is being developed by a team that self-organizes around its goals rather than a group of individuals working separately. Here are few rules of thumb to creating a team instead of just having a group of individuals:

  • Don’t assign tasks to individuals. Let the team figure out who works on what, and let the team take responsibility for all tasks rather than individuals. Everyone on the team needs to agree on what it means to be done and how, and everyone on the team needs to take responsibility for it.
  • Strive for cross-functional skillsets. Let the team members teach each other so they can better help each other in the future rather than allowing one person to become the bottleneck for any particular kind of task.
  • Try not to change team size or membership very much. A good team can be a fragile thing.
  • Throw out the notion of individual code ownership. The entire team owns the code.
  • Create more of a bullpen environment where the team members are all basically in the line of sight of one another and can just talk to one another about what they’re working on. This was tough, because it made us give up our individual offices, telecommuting, and deciding our own hours, but it was well worth it for the improvement to the quality of our work day. Almost anytime a developer has to use some means of communication other than his voice to talk with a teammate, you have unnecessarily wasted time.
  • Put QA personnel on the team so that testing can be can be on-going and so the team can more easily take on QA and testing as a team-wide responsibility.
  • The whole team needs to agree on what kind of work is necessary for each task before it is to be considered “done”. (coding standards, documentation standards, how rigorously they test, etc, etc)
  • The result of this was a highly capable team that could “swarm” to complete goals reliably and quickly. We actually developed a sort of sports team mentality that made this way of working genuinely fun, and helped us push ourselves to be better all the time. There are a lot of challenges to creating a great team that can inter-communicate openly and honestly, but it’s an amazing thing to behold when it happens.

Estimates need to come from developers.

This is can be very hard for some organizations to swallow, but developers are the ones that are most qualified to do estimations, and as professionals, they should be working diligently to improve their estimation capability. When a developer is responsible for estimation, that developer feels a sense of ownership about his work and will feel a vastly greater sense of responsibility to meet the estimate. Developers that have never learned how to estimate will not be good at it at first, but there’s no other way for them to get good at it than to have them do it. There are a few major things that can vastly help estimation too:

  • Always break work down into tiny chunks. We generally broke work up into chunks that were less than 3 days of effort. I’ve never seen a task that couldn’t be broken down into subtasks that are this small or smaller, if the developers are so inclined.
  • The team should take a look at bad estimates in retrospect and figure out where they went wrong. Recognize work that will be impossible to do due to lack of clarity from the business, or unmet dependencies and push to have those requirements met.
  • For highly speculative work, just commit to doing a proof-of-concept, or time-boxed research.
  • The more developers that are involved in the estimation process, the better. You’ll probably want to include QA/testers as well, because good ones have unique insights.

Context-switching should be avoided mercilessly.

This means that each developer needs the discipline to make sure he only works on one task at a time, that the team works on as few tasks at one time, and that developers need sustained uninterruptable timeboxes where the priority of work is not allowed to change.

Most people (especially non-developers) have no idea how much of a huge time-suck that context switching is, and few realize how seldom it’s actually worthwhile. Also if the team is tackling as few tasks as possible at once, if any task can be finished by having more developers on it at once, then that’s what the team should do rather than starting as many tasks as possible and working on them in parallel. This might sound counter intuitive, but with the team ultra-focussed on as few tasks as possible at once, they can get the totality completed faster.

Try to always improve.

The team must commit to continually improving. At regular intervals, the team should look at how things are going and pick a few actionable things to make their work better/faster/easier. You’d be surprised at the huge speed improvements that can come from this practice over time. If the team has a shared calendar, set up a reoccurring meeting at a regular interval to have this meeting, and be laser-focussed on deciding what improvement you want to achieve and how exactly you’ll try to achieve that. We’ve always set aside 1 hour per week, but your mileage may vary. Start every meeting by evaluating how successful the team was with the previous meeting’s goals.

If something is difficult or time-consuming, do it more often.

This sounds crazy to the uninitiated but there are two reasons for this:

  1. Doing difficult things more often will drive out new creativity in making them less difficult. Pro-tip: Automation is almost always the answer when the problem is testing (See TDD, BDD, ATDD) or deployment. (See Continuous Delivery)
  2. Many types of tasks get more difficult the longer they are postponed and so are always easier if you do them more often instead of letting the risk of complexity pile up. Integrating code into the main branch is one example (See Continuous Integration). Deploying is another (See Continuous Delivery).

Be merciless about achieving high quality

Over the long haul, speed comes from quality, so everyone on the team needs to make quality their job. Everyone needs to test, and everyone needs to think about how to continually make testing easier without cutting corners.

  • Have QA personnel on every team, and have them testing as the team works.
  • The whole team needs to agree on what level of quality is needed, and what quality metrics will be used to judge every check-in.
  • Perform a root-cause-analysis on each defect. Try to figure out how to avoid that class of defect in the future.

Do everything in tiny increments.

Tiny increments reduce the risk of compounding problems, and allow you to get feedback and correct your course more often.

  • break-down your work into tiny tasks
  • plan just-in-time and just enough to do work.
  • test as you go, instead of in huge batches
  • merge code to the mainline early and often.
  • deploy early and often.
  • don’t add anything to the system that you don’t need until you need it (YAGNI)

Get customer feedback early and often

Get customer feedback early and often to converge on the best possible solution as fast as possible by constantly correcting course instead of blindly following some long-running plan or doing unnecessary work. The customer’s priorities and vision of the product will change as he sees results, so it’s better to not find out about these changes after a lot of unnecessary work has been done.

  • Do work in priority order as dictated by the customer. This will allow them to see the aspects that they care about most as early as possible, and therefore help you converge on a unified vision of the product faster.
  • Work break-down is better done in vertical slices where the customer can get some amount of real value from each piece (or at least see some evidence of work) rather than having many non-user-facing tasks, where possible. This allows for more potential for feedback.
  • The work shouldn’t be considered done by anyone on the team until the customer, or a representative of the customer agrees that it’s done.
  • Constantly improve estimation skills so that they are generally more reliable so the customer knows the cost of the work he has requested and can make realistic decisions based on that.

Anyway, feel free to try what you want and throw out the rest!

One fine day in the trenches, a junior engineer was tasked with optimizing a particular web service call that was way too slow (sometimes slower than 20 seconds!). I had just figured out how to use a performance profiler for that particular platform, so I thought I’d offer to show him how to use it. His response was that he was on a tight deadline and didn’t have time to learn to use the profiler, and that he was just going to tune the database calls to make them perform better.

Later in the year when the problem still wasn’t solved, I had a chance to take it on myself. The profiler showed me that this service call was itself making around 20 web service calls to a second system that could obviously add up to 20 seconds, if that other web service took 1 second for each of its responses (and the logs showed that sometimes it did). It turned out that almost all of those web service calls were redundant and completely unnecessary. Simply removing them (which took about 1 day of effort) shaved 19 seconds off this web service call’s duration in production in some cases.

The total execution time spent on database calls was 5 milliseconds. A 50% improvement there would probably take a week’s worth of effort, and really only get you 2 or 3 milliseconds of performance improvement.

The moral of the story is that there’s a pretty simple process for optimizing a system:

  • Identify and isolate the component that is tying up the most time in the system (let’s call it the bottleneck)
  • Optimize that component to make it less wasteful
  • Repeat

Skipping step 1 isn’t a shortcut. It’s absolutely critical to know that you’re optimizing the bottleneck, and not some other random component.

This is pretty obvious stuff for most experienced developers of course. And while they seem to know to apply this to the systems that they work on, it’s more rare that they notice it outside of the code and see it in the process of software engineering in general.

The software development process is a system as well, made up of components. Requirements come in at one end, and deliverables go out the other end. As people who are involved in that process, we naturally want it to be more and more efficient. We are constantly trying to figure out how to optimize that system so we can deliver better software in less time.

So step 1: Identify and isolate the component that is tying up the most time in the system.

Not every team develops software the same way, so this component of the process could be different team by team. In practice though, I’ve always seen teams spending the most time doing these sorts of things:

  • Refactoring poorly written code so that features can be added
  • Learning/relearning existing code that has long since swapped out of all developers’ “mind pages” (for extension or bug-squashing).
  • Debugging
  • Performing Manual User Acceptance Testing (by both developers and testers)
  • issue tracking (administration of that tool and issue in it)
  • discovery and documentation of steps to reproduce
  • triage (what bugs to squash first?)
  • communication between customers, support, QA and developers
  • customer support
  • customer dissatisfaction leading to churn and reputation problems (affecting the entire organization)

All of these time consumers are affected directly or indirectly by our ability to control quality. In my experience, the majority of the software development process isn’t hung up in requirements analyis, design, or implementation. It’s hung up in all these Quality Control related issues. Quality Control is the constraint that keeps us from creating the right software faster. Steve McConnell confirms it in a more scientific way than I ever could:

The rest of us would do well to learn from a discovery made by IBM in the 1970s: Products with the lowest defect counts also have the shortest schedules (Jones 1991). Many organizations currently develop software with defect levels that give them longer schedules than necessary. After surveying about 4000 software projects, Capers Jones reported that poor quality was one of the most common reasons for schedule overruns (1994). He also reported that poor quality is implicated in close to half of all canceled projects. A Software Engineering Institute survey found that more than 60 percent of organizations assessed suffered from inadequate quality assurance (Kitson and Masters 1993). On the curve in Figure 1, those organizations are to the left of the 95-percent-removal line.

(http://www.stevemcconnell.com/articles/art04.htm — The whole article is extremely illuminating)

What this means is that if we really want to move quickly, we need to take a good hard look at how we manage quality. Of course this is counter-intuitive, and its actually hard to swallow, because ensuring quality can be mundane and tedious if you’re not doing it right. But as developers we’d be silly to ignore the real bottleneck in the process, and optimizing for that. Micro-optimizations elsewhere in the process will yield very little benefit compared to improving measures that ensure quality.

As developers, we love to argue about and discuss these micro-optimizations all day long:

It sure seems silly though to be pursuing micro-optimizations, at the expense of tackling the real #1 time-suck: Quality.

Prototyping is a design methodology that basically involves “trying stuff out and seeing what works”. It’s meant to be iterative in that you continue to create prototypes (possibly by alterring previous prototypes) until you’ve got all the information you need to say your design is complete.

If we were building cars, we would probably first prototype a body design with CGI to be able to collect feedback before building the car itself. It’s much cheaper (effort and materials wise) to iteratively develop the body with a 3d model than with the actual materials that would be used.

If we were designing websites, we might prototype the design in photoshop for a bunch of iterations until the client is happy. The great thing about that is that the web designer can actually chop up that protoype in photoshop, and use it for images as part of the work after the client has approved the design stage. In this case, the prototype isn’t a throw-away prototype — it’s actually useful to bring into the implementation stage.

In software development we can (and usually do) prototype similarly. There’s usually a point in the design process where a developer needs to “try some stuff out and see what works”. Once the developer has proven a methodology with code, the prototyping phase can mostly be called complete. Throwing away that code would widely be considered a huge waste though, so it usually ends up being used in the final implementation as well.

Prototyping VS. Hacking

It’s probably safe to say that many developers do prototyping (a design methodology) and implementation at the same time bouncing back and forth between the two as necessary, blurring that line between design and implementation.

This way of developing is seductive, because you get straight to banging on the keyboard, which feels like you’re making progress, but it’s really upstream in the design stage where you get the opportunity to save the most amount of time. If you don’t distinguish between design and implementation, it’s tough to tell which stage you’re in at any given time, and so it’s tough to determine if you’re in the right stage.

You can imagine how wasteful this back-and-forth would be if you were an automative engineer instead, and you swapped back and forth between prototyping and working on the implementation details. As you iterate on the prototype, those details you worked on in the last iteration could be completely removed — that’s a big waste of time, and it’s often emotionally difficult to let go of design decisions that have undergone a lot of implementation work. So while realistically design never really stops, you do want to do as much up front as possible.

So there are definitely right and wrong ways to prototype, but it’s extremely valuable and worthwhile if done correctly. I doubt many developers would argue that sometimes you just need to try stuff out and see what works. Very few developers are interested in development methodologies that omit the ability to prototype.

So What about TDD?

This seems to me to be the most often used argument against Test Driven Development (TDD). I agree that TDD doesn’t work well for tasks that don’t have clear-cut solutions, but TDD is still always applicable downstream after the solution has become more clear. The trick is to reach a certain level of solidity with your solution’s plan before you get into TDD.

Here’s a case in point as Ron Jeffries attempts to “solve sudoku” with TDD without even knowing the rules of the game. If you follow his method of solving the problem and compare it against Peter Norvig’s design-first approach, TDD starts to look like a miserable failure. Indeed, some have dragged this out as a case against the utility of TDD “in the real world”. Ron Jeffries’ example doesn’t really make the case against TDD at all though; It simply shows that you have no business being in the TDD stage before you fully understand the problem and you’ve done adequate design. TDD doesn’t preclude other design methods (like prototyping) at all.

And in cases of large architectural issues, prototyping is likely not the first tool that you should turn to — it’s too detail oriented, and you can iterate much faster on designs expressed via whiteboard doodling, flowcharts, UML, etc. You might move to prototyping at some stage later though before you completely decide on some particular design. The important part to keep in mind though is that you’re not in the implementation stage — you’re planning. Changes are cheap and easy at this stage, and it saves you so much more time than if you just jumped in and started developing.

The software that comes out of the prototyping process is a nice side effect, but it’s almost never ready for production. This is where TDD comes in. If you have a clear understanding of the solution you want to implement (whether you used prototyping or not) then your design is sufficiently complete to start TDD. In this way TDD and prototyping are not opposed at all. Prototyping is just one possible methodology in the design stage that should precede implementation. TDD just says that tests should be written before implementation. It doesn’t say anything about what else you do prior to implementation.

So, there’s nothing un-TDD about prototyping a solution, setting it aside, writing your tests, then writing an implementation that passes those tests (based on the knowledge you got from your prototype). Ideally you’re working with units that are single-minded and small so copying/pasting from the prototype isn’t even necessary, but to be honest, I think some copy/paste from the prototype is fine. If it’s done in a careful way where you’re not hurting the quality of the final code (because prototypes generally don’t care all that much about code quality), you might as well make use of your previous work, if you can. You just have to be careful that it’s the quality you want, but then again TDD will force a lot of the quality.

The Steel Thread

The concept of the steel thread is one that I use quite often on the team that I work in when we’re working on complex tasks. It’s definitely a loose sort of prototype, and it’s meant to prove the basics of our core concepts. Basically, we concentrate on the core functionality of the complex task, putting priority on the completion of the “happy path“, the path of execution without error scenarios, special cases, etc. Once that core functionality is in place, the general design can be considered verified, the necessary components can be easily identified, and the existing work can be evolved to something high-quality (possibly through a fresh start with TDD). Work is easily divided amoung teammates after this steal thread has been acheived.

Spike Solutions

Spike Solutions are like the opposite of the steel thread in many ways. Where the steal thread tries to prove the viability of the entire solution (in a breadth-first sort-of-way), a spike solution will try to prove the viability of a part of the solution that isn’t very well known, in a depth first sort-of-way. It’s useful when there’s a particular piece of the solution that we’re not sure will work, so we prototype that part as quickly as possible to make sure our overall solution won’t fail because of that single part.

The point of the concepts of steel threads and spikes is to tackle the unknowns early on and prove the viability of the entire solution as early as possible. Once everyone is relatively sure that the team is on the right path and the design is firmed up, you’re ready to get into the more detail-oriented aspects of development. It’s not until this point that I’d switch to TDD, and start writing my tests for the implementation stage of the work.

Is this just BDUF?

I don’t think so. I’m not proposing a huge documentation process that specifies every aspect of the final product. I’m simply talking about spending some time to loosely figure out what we’re going to do before we do it. It’s time well spent, because it’s just so much cheaper to make major changes at this earlier stage in the process. We’ll still figure out the minor details as we go.

“We will encourage you to develop the three great virtues of a programmer: laziness, impatience, and hubris.” — Larry Wall

In this quote Larry Wall makes some great points about what it means to be a great programmer in a zen-like, toungue-in-cheek sort-of-way. Let’s look at one aspect of the Virtue of Laziness.

Every reasonable developer knows that he should write his code to be easily maintained. As part of that work, it’s natural for that developer to want to try to make his code as ready for future change as possible. Taking that desire a few steps further, he might want to write code that solves future needs in addition to the current requirements, so that that code doesn’t even need to be revisited. Unfortunately, that’s a common mistake that leads to a lot of wasted time and less maintainable code.

I’d like to propose a principle of software development that I’ve come to believe:

The most maintainable code is the simplest possible code that meets all the requirements.

This is almost a tautology because simple code is (by definition) easier to understand, so it’s easier to change/enhance/extend. Code that does more than what’s currently necessary is not as simple as it could be though, so it’s not as maintainable given the current requirements.

One might argue that code that does more than what’s necessary can meet future requirements without change, thereby proving itself to be more easily maintained (no maintenance required!), but all of that hinges upon the developer’s ability to consistently and accurately foresee future requirements. Otherwise that developer has:

  • wasted time developing unnecessary functionality
  • decreased the maintainability of the codebase

Even when everybody is pretty certain of the future requirements…

Now let’s consider the case where the developer is almost absolutely sure that the requirements will change a certain way, and so the code should be written to handle that scenario as well. Wouldn’t it be better to do that work now? No. It still takes more time and effort (at a time when we know its not necessary), and it’s still less maintainable code until those requirements change (in case other unrelated changes are required). On top of all that, there’s still risk that the requirements won’t change in the anticipated way!

Even when everyone is absolutely 100% certain of the future requirements…

Now let’s consider the case where everyone is 100% certain that the requirements will change in a specific way, and you can write your code to handle that way as well as the current way. In that infrequent scenario, you don’t face a bunch of losing propositions, so it does make a bit more sense. However, I’d still like to pose a question: how much more work is it to do the additional work later instead of now? Truthfully, the amount of effort is virtually identical regardless of whether you do it now or later. That means you can enjoy maintaining the simpler version until the time comes that it needs to be changed to meet new requirements.

This isn’t a new idea I’ve dreamt up: it’s the YAGNI principle, and it’s an essential part of Uncle Bob’s 3 Rules of TDD. It’s also a great way to avoid unnecessarily complex code.

(It’s important to note too that YAGNI doesn’t say to avoid making your work as futureproof as possble. YAGNI just says not to put extra code into making that possible. For example, if you create a class for a UI button that will be used to save some user data in a form, don’t call it “UserDataSaveButton” — just call it “Button”. It’s no extra work and the re-usability possibilities have increased dramatically.)

We’re not engineers building bridges; we don’t work with unchangeable steel and concrete, and we can accomodate changing requirements easily. It’s unrealistic and wasteful to guess that we know what future requirements will come, especially when the potential work saved in the long run is neglible.

Here’s a controversial opinion, but one that I really believe to be true: most of the things we do to produce code faster are actually counter-productive. It sounds crazy, and I would naturally oppose it except that every time I ignore this lesson, experience has a way of reteaching me. This is as much a message to myself as it is to anyone else that might happen upon it (I’m talking to you, viagra comment spammers).

Development Time VS. Maintenance Time

There’s no doubt that the original development stage of a piece of software can be time-consuming. It’s interesting though, when you consider the difference in time spent in development vs. time spent in maintenance. At what point do you cross into maintenance mode?

Let’s face it: you’re actually also in maintenance mode pretty much right away, unless you’re some superhuman that conceives of the whole system at once and doesn’t make any design tweaks (or even typos) while coding. But you’re not. So the development-time VS. maintenance-time dichotomy is a false one. You’re always in maintenance mode.

Invest Early, Invest Often

So if you spend so much time (all your time) in maintenance mode, then it makes sense to do every little change and tweak in such a way that your subsequent tweaks and changes are as easy as possible. We should be spending the majority of our energy optimizing our time spent on maintenance and not our time spent on development.

Now it’s natural that one might say “well why not optimize both?”, and that’s a valid question. There’s no good reason not to try to optimize both, but we have to remember to never ever ever optimize the development process at the expense of the maintenance process, simply because we just spend so much time in maintenance mode.

Some examples

Skipping the planning stage.

This is generally caused by the “Weeks of coding can save days of planning” fallacy. Uncover all the main parts of your design before you code yourself into a corner that requires a near-rewrite to escape.

Writing untestable code or skipping unit-testing

Unit-tests are your insurance that future changes don’t cause non-obvious regressions. They also have a magical way of creating better factored, higher-quality code. If you skip this stage, your code will not be factored as well as it could be, and come maintenance time, you’ll be scared to make changes for fear of breaking something somewhere else.

Writing classes and functions that do a lot

This one is a really seductive one. We want to be consumers of classes and methods that do a lot for us, because higher levels of abstraction are how we get more done in less time. From the consumer stand-point, this makes lots of sense, and it’s actually the best way to improve development speed and maintenance speed at the same time. If you’re also the person writing the class or function that does more though, you have to be very careful to use the highest abstractions possible for that as well, even if you’re writing its inner abstractions as well. What this means is that every level of abstraction does as little as possible, and you build up to the level of abstraction that you want with those thin, thin layers. Classes and functions that try to do a lot instead of relying on other classes and functions are very hard to maintain, simply because there’s so much to do read and understand, and there’s no easy way for the maintainer to skim and decide what piece is relevant, because the pieces are doing multiple things.

Leaving code in a low-quality state

I’m talking about everything involving non-readable and poorly factored code, including bad names, foggy separation of concerns, etc. Code should not be written just for the machine. It needs to be written for the next guy that comes along as well. That next guy could be you. Or a violent psychopath

Using code generation

Code generation is just pure evil, if there’s ever a remote chance a human will ever have to read it or change it. A computer program doesn’t know how to write code that a human will understand, and human understanding is exactly what we need to strive for if we care about making maintainable code.

Understatement of the day: You’re more than just a typist

It’s largely irrelevant if you can touch type at 90 wpm. It’s largely irrelevant if your IDE can do automated refactors. It’s largely irrelevant if your efficiency with Emacs or vim makes you look like Neo in The Matrix. These things are great bonuses, but won’t make much difference to your proficiency (and efficiency) as a programmer if you neglect to first consider the maintainability of your codebase. (Corollary: In the name of all that is good in this world, please don’t name your variables p1, atb, etc, or your functions like atoi() just for the sake of saving keystrokes!)

Time and time again my mistakes remind me: there’s only one way to be more productive over the long haul: work with better, higher abstractions.