We're in the Stone Age of Software Engineering Management

Obviously from the title, I think software engineering management is in a Stone Age. Before I get into my arguments though, I'd like to say that this isn't really about any particular manager or managers that I've had in the past. It's really about counter-productive patterns that I've seen that I really think we need to evolve from. I've also been responsible for some of these mistakes myself as a manager or technical lead over my years in software engineering, so if this comes off preachy, that's not the intention.

There are a few common problems in engineering management that keep us in the Stone Age though, and I'd like to detail some of them.

The longer you manage, the less technically competent you become.

"Technology is dominated by two types of people: those who understand what they do not manage, and those who manage what they do not understand." -- Putt's Law

It's undeniable that knowledge work has some fundamental differences from factory work: there's never a point where you've completely learned how to do your job, and in fact the knowledge required around your job is constantly changing. For this reason, the longer you're not developing software, the worse you get at it. Being able to type code is only tangentially necessary for software development. The real job is making decisions about how the software should work.

This is one of the few cases in industrialized production where the worker quite regularly knows more about their job than their manager. And if the manager does happen to know more, the gap between them will be constantly narrowing.

The impact of this fact is pretty wide-reaching. It means that a manager is generally not going to be useful for their technical competence, or at least that their usefulness will be waning. In my career I have rarely met exceptions to this rule.

Management is viewed as a promotion and not a distinct role with a separate set of skills.

I'd like to propose instead that management is a distinct role that requires a different set of skills, none of which require the ability to make technical decisions (and I'll get to those later).

Unfortunately management in the software development industry is looked at much like management in other industrialized production. Management is widely considered to be "a promotion" that developers should aspire to. It's a common expectation that developers at a certain level of seniority (or age) should be moving into management. There's a very real pressure from the rest of the world too because they view management as a "higher" position, so you often see developers pushing for this "promotion", even though they have not acquired the disparate skill-set required of a manager. The result is that often companies trade excellent developers for terrible managers, and often to the detriment of those developers as well.

Management is viewed as an imperative to command and control.

It's common to move people into management based on technical ability, because surely we must need a technical person to make the important technical decisions, right? That's certainly the argument that's most often made to support the management-as-promotion mindset.

There are huge drawbacks to the manager being "The Decider" in technical matters though (notwithstanding their continually eroding technical ability):

It certainly doesn't have to be this way, but many view management not just as a chance to command and control, but also as a responsibility to command and control. If the team is allowed to make technical decisions together though, on their own, all of these particular problems can just disappear. And I would argue that if a manager doesn't have a team that can be trusted with this responsibility, it is the manager's responsibility to transform that team.

Interestingly, it's a pretty normal refrain from engineers that they want their manager to be technical. I believe that much of that sentiment comes from an assumption that management is command-and-control by definition, and of course the worst of both worlds would be a non-technical command-and-control manager (and I whole-heartedly agree). I also think that if management would instead trust technical decisions to those most close to the work, engineers would be far less likely to care about their manager's technical competence.

Management focuses on enforcing the predictability of software development rather than on mitigating its undeniable unpredictability.

Software development is development of complex systems that are, by definition, extremely difficult to predict. Typically management tries to predict, control, and reprimand. Given the fact that complex systems are resistant to those types of actions, the effort is largely fruitless and often counter-productive.

Some examples:

Estimates from developers, when treated as self-imposed deadlines, are really no better than deadlines, because prediction of how long a software project will take is so incredibly difficult and developers are rarely given the training or time to do better estimates. Estimates like this end up being another mechanism of control, but a somewhat more insidious one, because developers will feel like they have no one to blame but themselves for "missed estimates".

All of these management efforts are time-consuming, stressful, and mostly counter-productive (in that they take away from the developer's time to write software).

Of course you might reasonably ask how the feasibility of a proposed feature can be assessed without some estimation of its costs, which is a fair question, but an effective manager must realize that the estimate (as well as the scope of the work being estimated) must be continually revisited and revised and the stake holders need to have their expectations continually managed/adjusted.

The complex system that the developers create includes all the people involved.

Anytime something happens in the system, it can't realistically be viewed solely from a technical perspective. The technical aspects exist in large part because of what developers did or did not do to make them happen. The interdependencies between people and the technology forms in such a way that most mental models that separate them are not grounded in reality. There are a few results of this:

The complex system that the developers create has a lifetime.

As a piece of software is used more and more over time, the chances of a user finding a way to put the software in an unexpected state increases. It's even common for new bugs to arise in narrowly scoped software that's over 20 years old.

Ultimately the only software that doesn't require maintenance is software that no one uses anymore. Otherwise, there's no such thing as completion of a piece of software. At best you can get a convergence toward completion, assuming the project is narrowly scoped and you can resist feature-creep.

Unfortunately management typically looks at software development as a series of projects or features with definite endings where there is an expectation that no more work will be necessary at the completion of the "project", and that the team will be 100% available for the next project.

This fallacy may or may not lead to completely unrealistic expectations on the first project or two, but as the number of projects "completed" increases over time, the team will become busier and busier maintaining those projects and the team will necessarily get slower and slower.

There are innumerable counter-measures that can be taken to ease the maintenance of software over time of course, but most of these require management to first realize that this reality exists, and to allow the team to spend the time to take these counter-measures.

Management that doesn't realize this often misses the point of spending time on anything that doesn't yield value immediately. For example, automated tests are in many scenarios a super valuable way of ensuring regressions don't creep in. They often don't provide enough value to offset their immediate up-front costs, but I've rarely seen them not be a net positive over time as they eliminate the need for slower manual testing forever after. Short-sighted management will be reluctant to make investments like this, and therefore doom the team to lower productivity over the long haul.

Goals and priorities are rarely clearly and intelligently set.

Goal and priority setting is the absolute number one deliverable that management has for the team. Unfortunately it is common for management to busy itself with other less productive tasks, often involving micro-management, and to actively disrupt the team's effort to ensure that goals and priorities are clear.

Some common ways that management fails at this:

The most common failure I've seen here is management's unwillingness to choose a more important goal between two top goals. It's fine for management to ask for help from the team in that prioritization, or even for management to pick one randomly (if they're really so equal, it shouldn't matter), but what is unacceptable is for management to say "These are our two #1 goals". In that case, the team is forced to take over this management responsibility, because any one person can only do one thing at a time. If the team has not learned to manage that responsibility themselves, they will often be terrible at it. Some members will be working on one thing and others will be working on the other when just a touch of management could have had them coordinating to get one thing done first and delivering value as soon as possible. Instead the manager in these scenarios has ensured that value will be delivered in a slower way than what's optimal.

Efforts and Practices are rarely critically examined with any attention to the results.

It's unfortunately extremely uncommon for managers today to pay much attention to the actual results of the team's effort and the practices that it follows.

I think that a lot of the reason behind this is that it's extremely difficult to admit that things aren't going well when we've tied those results to our self-image. The only thing worse than mistakes though are mistakes that go uncorrected. Unfortunately the first step to correcting a mistake is to admit it exists.

Ego can also get in the way and take management off into more interesting (or brag-worthy) endeavours than what's best for the business. I've seen this happen countless times where teams deliver absolutely nothing while working on the latest in tech. If the same ego-driven management style reaches high enough into the organization, and the company is profitable enough to support it, it's easy for a team to get away with this blunder for years (which I've also seen).

Copy-cat or Cargo-cult management is probably the next most common excuse for not examining results. Often I've heard "google does x" or "y is an industry best practice" or even "that's not Agile" without any discussion of whether a particular practice makes sense for the team's own particular goals, scenario, or needs. Often these managers feel that they're adhering to "best practices" and so the actual results will necessarily be optimal and won't need to be examined. I'm definitely a proponent of agile methodologies in general, but saying whether or not something is "Agile" explains nothing about it's utility to the organization. There are no sacred practices that should be allowed to escape scrutiny. Any practice that can't be shown to be adding value should be canceled immediately.

It's common as well when a manager does manage to pay attention to results that his/her methods are extremely flawed. Probably the most infamous example of this is how managers used to count lines of code to judge the productivity of a developer. Counting the number of commits is also ridiculously flawed in the same way: they're missing what it means to be productive in an engineering context entirely. The least productive engineers are doing things like building needlessly elaborate architectures and suffering from NIH when they could just use a far better 3rd party library. Yes, you do need to be able to determine if a developer is productive or not, but that doesn't mean that that determination can be quantified. There will be some very important things that you need to measure that aren't quantifiable, so managers need to be comfortable with qualitative measurements.

In short, I think it's pretty common for engineering management to be actively harmful to their team's speed and their product's quality. Because uncorrected management mistakes impact entire teams, it's quite easy for a manager to have an overall net negative contribution to the organization. Of course it doesn't need to be like this, but a lot of the common expectations of management will really need to change first.

Tags: management

← Back home