Why Does Bad Waterfall Happen to Good People?

I’ve recently seen a case of spontaneous waterfall-style development emerging where the team was generally empowered to avoid it or at least to correct against it, and certainly wanted to, and yet did not. From the outside it seemed like waterfall was a kind of steady-state for them that they’d trended toward and didn’t have the escape-velocity to move toward anything else. They weren’t happy with it either. Most of them felt quite alienated from their work which is a common result of waterfall development. What could have happened?

I think there are a bunch of properties that can lead to waterfall-style development and management is only one of them. Before I get into those, I’d first like to propose a simplified explanation of what it means to be waterfall.

What’s This Waterfall Thing That They’re Talking About?

Waterfall development is development with varying degrees of one-way hand-offs between multiple disciplines. The best way to think of it is to imagine a sort of human assembly line where each homogenous set of disciplinarians pass the result of their discipline down to the next homogenous set of disciplinarians. More concretely, you’ve got these hand-offs when the product managers tell the designers what to design and the designers tell the engineers what to build and the engineers give the testers the work to test and the testers give the release manager a release to release and the release manager gives the sysadmins a deployable to deploy. It’s all very well organized and simple to explain and generally soul-sucking and totally inefficient in practice.

It’s soul-sucking for a few reasons:

  • The downstream disciplines have very little power to make drastic changes, because of the tyranny of sunk-cost fallacy. For example, if the testers find some behaviour that is terrible for the users, the engineers will shrug and say “not a bug… works as designed!”. Or if the engineers determine there’s a performance problem with the fully completed hi-fi design mocks that make them infeasible, the designer is at best forced to go back to the drawing board to take another shot at getting an idea that the engineers might agree on.
  • The downstream disciplines have no idea why they’re designing/building/testing/releasing what they’re designing/building/testing/releasing. Waterfall is set up as an assembly line in the name of efficiency and that means that downstream disciplines are on a need-to-know basis. The problem with this is that you suck the motivation out of everyone because knowing “why” is precisely where motivation comes from.
  • The upstream disciplines are generally resented by the downstream disciplines for how they’re always ignoring the important downstream concerns.
  • The upstream disciplines are constantly pressured to do more and more flawless work “to minimize all that expensive back-and-forth”.

It’s funny when people start to talk about “all that expensive back-and-forth” because trying to avoid that is precisely what makes it so expensive. The upstream disciplines are always expected to trend toward perfection in pursuit of the ideal one-way waterfall. When you hear the engineers asking for perfect hi-fi mocks before they start their work, you’ll know you’re approaching waterfall-land. Your product development flow is about to go into slow-motion.

The Role of Single-Discipline Teams

There’s an idea too that similar disciplinarians should be the ones working and managed together as a unit; that’s how you achieve consistency in the discipline and how you ensure these specialists are always pushing each other to greater heights in their respective crafts. Unfortunately these specialists never really get a full view of the entire process. With neither broad understanding nor a holistic ownership mindset, the end result is that you’re paying a whole lot of smart people while simultaneously trying to avoid them thinking about the business’ problems. A specialist that doesn’t fully understand the business goals or how software development works in general will always be a pretty weak specialist regardless of the depth of their specialization.

This is how you get dysfunctional scenarios like:

  • QA personnel that feel like their primary goal is to try to find a way to block the release.
  • Designers that feel like customers should wait for the perfect aesthetics before we can deliver new functionality.
  • Engineers that can’t make good decisions about when to take on technical debt and when to pay it down.

Having individuals confined solely to their area of expertise can make sense in the short term! For example, I’m an engineer and I’ve regularly had terrible product ideas. In the short term there’s no sense in involving me further “upstream” in the process; I’ll only drag it down.

In the long term though, my terrible product sense is certainly a liability. And on a daily basis when I stand in the shower or on the subway thinking about work, I’m just coming up with more terrible ideas. The better thing to do is to teach me about the business and its goals and make that investment in me. You’ll occasionally get a worthwhile idea from me, and I’ll always be more willing to go that extra mile because I know why I’m building what I’m building and that it’s worthwhile work.

The Role of Product Management

Product management can play a significant role in whether or not that teaching and that context-sharing happens. I’ve seen teams prioritize their own product backlog for multiple weeks of work without a product manager present — That’s a team that the product manager has really invested in and the result is that the product manager is comfortable giving them agency. There are some unexpected things that can happen here: the team will often have better ideas or prioritizations than the PM, or at the very least will have top-of-stream improvements on the PM’s ideas that would never happen otherwise. When this is really working, the one brain of the PM will almost certainly not be able to beat the multiple brains on the team. And when the PM considers themselves a contributing member of the team, the team will necessarily outperform the lone PM.

There’s a simple but slow progression that a skillful PM can push the team through if they’re in waterfall-mode due to the product management stage:

  1. Allow the team to own the solutioning. Take the team from working on predetermined solutions and start to instead involve them in determining the solutions themselves. Achieving this really takes a considerable amount of effort, but it usually pays for itself in spades later with the PM’s freed-up time, better work prioritizations, better product decisions, and less process churn (“expensive back-and-forth”). It’s amazingly motivating for the team too.
  2. Allow the team to own the problem-selection. Once you’re comfortable with their performance in determining the solutions, start to get their input on what the problems to solve are, and coach them to seek problems that meet the team’s goals and show them to independently collaborate with other sources of company/product/user information. Give them access to the metrics they’re trying to improve. In time, be open to letting them try some ideas that you might not think are great; at the very least it’ll be a learning.
  3. Allow the team to have input into the mission. Everything is easier when the team thinks it’s working towards the right goals. The easiest way to get the team working on the right goals is to ensure that they have that context. If you brought the team through steps 1 and 2 above, you can see that they’re much more bought-in to plans that they’re part of. Involve them in actually determining the mission too! The end result will be a team that is incredibly fast-moving and energized.

Not to be too deep, but even though this is framed here as a maturity progression for the team, it can also be a maturity progression for the PM. The PM needs to be able to become the team’s guide to the market, the user’s needs, the company’s goals, etc, etc, instead of playing the part of puppeteer. And we’ve seen that it absolutely doesn’t put the PM out of a job. There’s definitely more than enough work in ensuring that you can be expert guide and a strong team mate. The product manager on my current team was brave enough to engage with the rest of the team like this and the results are really amazing. Instead of pushing product management on the team, he pulls the team into product management.

Also it should be obvious that when the team is spending time on things that are not software production that they will almost certainly release less lines-of-code. I’ve found instead though that the products developed this way are generally always delivering more value to the user, and doing so at a faster pace. No user is asking for “more software”.

Product Management Is Just One Possible Contributing Factor Though

In fact any of the disciplines on the team has the ability to cause the emergence of a waterfall model of development. It’s a systematic problem that is really tough to solve because it emerges from the socio-technical complex system of the software, the team, the company, and the market. As a result, there’s almost never just a single cause of it.

So with the obvious influence of product management aside, these additional properties seem to make waterfall vastly more likely:

  • Inability to see how traditionally downstream disciplines can actually occur simultaneously or even upstream.
  • Perfectionism (release-o-phobia) due to the perception that a particular stage is very expensive or irreversible.
  • An “Only specialists can do X” attitude (often due to perfectionism), sometimes worsened by an approval process requiring sign-off.
  • Lack of will to collaborate across disciplines. There’s an inertia to collaboration that sometimes is hard to overcome.
  • Cargo cult adherence (eg “Such-and-such company does it this way”, “This is the way I’ve always done it”, “My software engineering textbook from 1968 says this is how you do it”)

I’ve personally never seen waterfall-style development occur in the absence of these factors so I think about ways to stamp them out whenever I see them.

For the team that just doesn’t know how to defeat waterfall but wants to, there are a tonne of things to try:

  • Create an environment of collaboration. Usually this involves collocated cross-discipline teams. When other companies are saying “Design is crucial. We should have a design team.”, try to instead say “Design is crucial. We should have a designer on every team.” It’s this attitude that will have your team able to deliver software entirely on its own without endless meetings with other teams.
  • Make sure the team is surrounded by the information that it needs to be to make the right decisions. They should have easy access to your user-researchers, data scientists, customer-support, subject-matter experts, etc, etc to easily learn everything that they have to about their problem space. If you can put those specialists on the team as well, that’s even better.
  • Try to get the team to focus on a very small number of things at once, ideally 1. You want to focus on finishing instead of starting and getting each member of the team to do something different is the opposite of that.

Once you’ve achieved that proper environment, experiment with ways to disassemble the traditional waterfall order of things:

  • Have many people involved in all manner of planning (if they’re interested… Some teammates may prefer to trust the others in their decisions and avoid all those meetings).
  • Have a common set of standard UI components so engineers can assemble pretty good designs quickly on their own.
  • Practice devops! It’s the ultimate collaboration of developers, release managers and sysadmins often with all 3 roles rolled into one person (and heavily automated!) but at least on the same team.
  • Decouple release from deployment with feature flags and actually deploy before design is even complete. Often you can deploy all kinds of non-production-ready stuff constantly and only turn it on when it’s ready. That’s the essence of Continuous Delivery
  • Do testing BEFORE engineering. TDD and BDD are possibilities. If you have QA people, let them advise on what kinds of tests would be useful. That’s what Quality Assurance actually means.
  • If possible, have everyone own testing. If possible, have engineers own test automation and test their own/each other’s stuff.
  • Have lo-fi designs that are good enough to start engineering with. Let the hi-fi designs come later… maybe much later; Depending on your situation, you might be able to launch to 1% of your users with lo-fi designs and validate the product before investing in higher fidelity designs.
  • Have engineers and testers think through the lo-fi versions of the designs with the designer. Treat the designer as the team’s expert guide rather than putting all the design responsibility on them.
  • Break up your work into smaller and smaller pieces. Even if you’re doing some hand-offs, rework on small pieces is much cheaper.

The greater goal is to value results over effort, and to build the right software rather than the most software. Getting as many efforts in progress at once is an anti-pattern. Making sure everyone has something to do is a non-goal.

Without that perspective, many of these practices will seem inefficient. Sure sometimes you’ve got multiple people working slightly outside of their specialty, but the resulting software is generally always better, and all that expensive back-and-forth and rework is virtually eliminated. People are generally happier and more engaged as well because they know how important the work is and why.

Special thanks to Taylor Rogalski for feedback!