I've recently seen a case of spontaneous waterfall-style development emerging where the team was generally empowered to avoid it or at least to correct against it, and certainly wanted to, and yet did not. From the outside it seemed like waterfall was a kind of steady-state for them that they'd trended toward and didn't have the escape-velocity to move toward anything else. They weren't happy with it either. Most of them felt quite alienated from their work which is a common result of waterfall development. What could have happened?
I think there are a bunch of properties that can lead to waterfall-style development and management is only one of them. Before I get into those, I'd first like to propose a simplified explanation of what it means to be waterfall.
Waterfall development is development with varying degrees of one-way hand-offs between multiple disciplines. The best way to think of it is to imagine a sort of human assembly line where each homogenous set of disciplinarians pass the result of their discipline down to the next homogenous set of disciplinarians. More concretely, you've got these hand-offs when the product managers tell the designers what to design and the designers tell the engineers what to build and the engineers give the testers the work to test and the testers give the release manager a release to release and the release manager gives the sysadmins a deployable to deploy. It's all very well organized and simple to explain and generally soul-sucking and totally inefficient in practice.
It's soul-sucking for a few reasons:
It's funny when people start to talk about "all that expensive back-and-forth" because trying to avoid that is precisely what makes it so expensive. The upstream disciplines are always expected to trend toward perfection in pursuit of the ideal one-way waterfall. When you hear the engineers asking for perfect hi-fi mocks before they start their work, you'll know you're approaching waterfall-land. Your product development flow is about to go into slow-motion.
There's an idea too that similar disciplinarians should be the ones working and managed together as a unit; that's how you achieve consistency in the discipline and how you ensure these specialists are always pushing each other to greater heights in their respective crafts. Unfortunately these specialists never really get a full view of the entire process. With neither broad understanding nor a holistic ownership mindset, the end result is that you’re paying a whole lot of smart people while simultaneously trying to avoid them thinking about the business' problems. A specialist that doesn't fully understand the business goals or how software development works in general will always be a pretty weak specialist regardless of the depth of their specialization.
This is how you get dysfunctional scenarios like:
Having individuals confined solely to their area of expertise can make sense in the short term! For example, I'm an engineer and I've regularly had terrible product ideas. In the short term there's no sense in involving me further "upstream" in the process; I'll only drag it down.
In the long term though, my terrible product sense is certainly a liability. And on a daily basis when I stand in the shower or on the subway thinking about work, I'm just coming up with more terrible ideas. The better thing to do is to teach me about the business and its goals and make that investment in me. You'll occasionally get a worthwhile idea from me, and I'll always be more willing to go that extra mile because I know why I'm building what I'm building and that it's worthwhile work.
Product management can play a significant role in whether or not that teaching and that context-sharing happens. I’ve seen teams prioritize their own product backlog for multiple weeks of work without a product manager present -- That’s a team that the product manager has really invested in and the result is that the product manager is comfortable giving them agency. There are some unexpected things that can happen here: the team will often have better ideas or prioritizations than the PM, or at the very least will have top-of-stream improvements on the PM’s ideas that would never happen otherwise. When this is really working, the one brain of the PM will almost certainly not be able to beat the multiple brains on the team. And when the PM considers themselves a contributing member of the team, the team will necessarily outperform the lone PM.
There’s a simple but slow progression that a skillful PM can push the team through if they’re in waterfall-mode due to the product management stage:
Not to be too deep, but even though this is framed here as a maturity progression for the team, it can also be a maturity progression for the PM. The PM needs to be able to become the team’s guide to the market, the user’s needs, the company’s goals, etc, etc, instead of playing the part of puppeteer. And we’ve seen that it absolutely doesn’t put the PM out of a job. There’s definitely more than enough work in ensuring that you can be expert guide and a strong team mate. The product manager on my current team was brave enough to engage with the rest of the team like this and the results are really amazing. Instead of pushing product management on the team, he pulls the team into product management.
Also it should be obvious that when the team is spending time on things that are not software production that they will almost certainly release less lines-of-code. I’ve found instead though that the products developed this way are generally always delivering more value to the user, and doing so at a faster pace. No user is asking for “more software”.
In fact any of the disciplines on the team has the ability to cause the emergence of a waterfall model of development. It’s a systematic problem that is really tough to solve because it emerges from the socio-technical complex system of the software, the team, the company, and the market. As a result, there’s almost never just a single cause of it.
So with the obvious influence of product management aside, these additional properties seem to make waterfall vastly more likely:
I've personally never seen waterfall-style development occur in the absence of these factors so I think about ways to stamp them out whenever I see them.
For the team that just doesn't know how to defeat waterfall but wants to, there are a tonne of things to try:
Once you've achieved that proper environment, experiment with ways to disassemble the traditional waterfall order of things:
The greater goal is to value results over effort, and to build the right software rather than the most software. Getting as many efforts in progress at once is an anti-pattern. Making sure everyone has something to do is a non-goal.
Without that perspective, many of these practices will seem inefficient. Sure sometimes you've got multiple people working slightly outside of their specialty, but the resulting software is generally always better, and all that expensive back-and-forth and rework is virtually eliminated. People are generally happier and more engaged as well because they know how important the work is and why.
Special thanks to Taylor Rogalski for feedback!