Obviously from the title, I think software engineering management is in a Stone Age. Before I get into my arguments though, I'd like to say that this isn't really about any particular manager or managers that I've had in the past. It's really about counter-productive patterns that I've seen that I really think we need to evolve from. I've also been responsible for some of these mistakes myself as a manager or technical lead over my years in software engineering, so if this comes off preachy, that's not the intention.
There are a few common problems in engineering management that keep us in the Stone Age though, and I'd like to detail some of them.
"Technology is dominated by two types of people: those who understand what they do not manage, and those who manage what they do not understand." -- Putt's Law
It's undeniable that knowledge work has some fundamental differences from factory work: there's never a point where you've completely learned how to do your job, and in fact the knowledge required around your job is constantly changing. For this reason, the longer you're not developing software, the worse you get at it. Being able to type code is only tangentially necessary for software development. The real job is making decisions about how the software should work.
This is one of the few cases in industrialized production where the worker quite regularly knows more about their job than their manager. And if the manager does happen to know more, the gap between them will be constantly narrowing.
The impact of this fact is pretty wide-reaching. It means that a manager is generally not going to be useful for their technical competence, or at least that their usefulness will be waning. In my career I have rarely met exceptions to this rule.
I'd like to propose instead that management is a distinct role that requires a different set of skills, none of which require the ability to make technical decisions (and I'll get to those later).
Unfortunately management in the software development industry is looked at much like management in other industrialized production. Management is widely considered to be "a promotion" that developers should aspire to. It's a common expectation that developers at a certain level of seniority (or age) should be moving into management. There's a very real pressure from the rest of the world too because they view management as a "higher" position, so you often see developers pushing for this "promotion", even though they have not acquired the disparate skill-set required of a manager. The result is that often companies trade excellent developers for terrible managers, and often to the detriment of those developers as well.
It's common to move people into management based on technical ability, because surely we must need a technical person to make the important technical decisions, right? That's certainly the argument that's most often made to support the management-as-promotion mindset.
There are huge drawbacks to the manager being "The Decider" in technical matters though (notwithstanding their continually eroding technical ability):
It certainly doesn't have to be this way, but many view management not just as a chance to command and control, but also as a responsibility to command and control. If the team is allowed to make technical decisions together though, on their own, all of these particular problems can just disappear. And I would argue that if a manager doesn't have a team that can be trusted with this responsibility, it is the manager's responsibility to transform that team.
Interestingly, it's a pretty normal refrain from engineers that they want their manager to be technical. I believe that much of that sentiment comes from an assumption that management is command-and-control by definition, and of course the worst of both worlds would be a non-technical command-and-control manager (and I whole-heartedly agree). I also think that if management would instead trust technical decisions to those most close to the work, engineers would be far less likely to care about their manager's technical competence.
Software development is development of complex systems that are, by definition, extremely difficult to predict. Typically management tries to predict, control, and reprimand. Given the fact that complex systems are resistant to those types of actions, the effort is largely fruitless and often counter-productive.
Some examples:
Deadlines are one such mechanism of control. Despite the fact that engineers are the best people from which to get accurate estimates, it is extremely common for managers to dictate deadlines for fixed-sized projects, and to repeatedly find their team failing to meet them. For some reason, this repeated failure is not an indication that something has gone wrong management-wise. Often the team is chastized, over-worked, or reprimanded instead. Of course the real world will have dates after which delivery of software greatly diminishes its value, but deadlines are simply a thoughtless and reckless way to enforce either minimizing a feature or lowering the level of quality that goes into it (usually the latter). When a business uses deadlines with a fixed scope of work and quality, it's willfully ignoring the reality of its team's capability.
Estimation is an example of attempting prediction. Complex systems are inherently difficult to predict as well, and we've all seen evidence of this in our own inaccurate estimates. The best success we've had with improving estimation is to limit the amount of work we bundle up in an estimate which is to say we've improved estimation by estimating less. The sum of a project's timeframe often ends up being greater than the sum of the timeframes of its parts though, because we often even fail to predict all the pieces of work that are necessary.
Estimates from developers, when treated as self-imposed deadlines, are really no better than deadlines, because prediction of how long a software project will take is so incredibly difficult and developers are rarely given the training or time to do better estimates. Estimates like this end up being another mechanism of control, but a somewhat more insidious one, because developers will feel like they have no one to blame but themselves for "missed estimates".
All of these management efforts are time-consuming, stressful, and mostly counter-productive (in that they take away from the developer's time to write software).
Of course you might reasonably ask how the feasibility of a proposed feature can be assessed without some estimation of its costs, which is a fair question, but an effective manager must realize that the estimate (as well as the scope of the work being estimated) must be continually revisited and revised and the stake holders need to have their expectations continually managed/adjusted.
Anytime something happens in the system, it can't realistically be viewed solely from a technical perspective. The technical aspects exist in large part because of what developers did or did not do to make them happen. The interdependencies between people and the technology forms in such a way that most mental models that separate them are not grounded in reality. There are a few results of this:
Developers can't simply be swapped in and out of the team without significant costs. They form relationships and interdependencies with both the technology and the other people on the team that are time-consuming and difficult to reform. This cost can be minimized by having the team regularly push to make itself pan-functional (usually via intra-team mentoring).
Conversely, a developer can have real negative impact on this socio-technological system in many ways, the most detrimental of which is by not acting in a way that's worthy of the team's trust. If a team member regularly refuses (explicitly or implicitly) to act in accordance with the team's general wishes, that developer must be removed, regardless of some abstract notion of "technical skill level" or how "indispensable" they've made themselves.
The team itself requires maintenance. They need to be encouraged and allowed to regularly take the time to look at their process and practices and decide what steps they should take to improve them. They need to be supported in their efforts to take those steps.
A high performing engineering team is more valuable than the sum of its highly performing individuals. A team needs to be cultivated carefully with an eye on creating the desired culture of collaboration and trust. This is extremely difficult and takes a great deal of time and effort, but it pays off wildly with much smarter decisions and much faster development.
Software development cannot be broken into an assembly line with specification, coding, and testing stages. All of these things occur and reoccur as necessary. They are all "development" and there's nothing to gain by pretending that the task is divisible.
As a piece of software is used more and more over time, the chances of a user finding a way to put the software in an unexpected state increases. It's even common for new bugs to arise in narrowly scoped software that's over 20 years old.
Ultimately the only software that doesn't require maintenance is software that no one uses anymore. Otherwise, there's no such thing as completion of a piece of software. At best you can get a convergence toward completion, assuming the project is narrowly scoped and you can resist feature-creep.
Unfortunately management typically looks at software development as a series of projects or features with definite endings where there is an expectation that no more work will be necessary at the completion of the "project", and that the team will be 100% available for the next project.
This fallacy may or may not lead to completely unrealistic expectations on the first project or two, but as the number of projects "completed" increases over time, the team will become busier and busier maintaining those projects and the team will necessarily get slower and slower.
There are innumerable counter-measures that can be taken to ease the maintenance of software over time of course, but most of these require management to first realize that this reality exists, and to allow the team to spend the time to take these counter-measures.
Management that doesn't realize this often misses the point of spending time on anything that doesn't yield value immediately. For example, automated tests are in many scenarios a super valuable way of ensuring regressions don't creep in. They often don't provide enough value to offset their immediate up-front costs, but I've rarely seen them not be a net positive over time as they eliminate the need for slower manual testing forever after. Short-sighted management will be reluctant to make investments like this, and therefore doom the team to lower productivity over the long haul.
Goal and priority setting is the absolute number one deliverable that management has for the team. Unfortunately it is common for management to busy itself with other less productive tasks, often involving micro-management, and to actively disrupt the team's effort to ensure that goals and priorities are clear.
Some common ways that management fails at this:
The most common failure I've seen here is management's unwillingness to choose a more important goal between two top goals. It's fine for management to ask for help from the team in that prioritization, or even for management to pick one randomly (if they're really so equal, it shouldn't matter), but what is unacceptable is for management to say "These are our two #1 goals". In that case, the team is forced to take over this management responsibility, because any one person can only do one thing at a time. If the team has not learned to manage that responsibility themselves, they will often be terrible at it. Some members will be working on one thing and others will be working on the other when just a touch of management could have had them coordinating to get one thing done first and delivering value as soon as possible. Instead the manager in these scenarios has ensured that value will be delivered in a slower way than what's optimal.
It's unfortunately extremely uncommon for managers today to pay much attention to the actual results of the team's effort and the practices that it follows.
I think that a lot of the reason behind this is that it's extremely difficult to admit that things aren't going well when we've tied those results to our self-image. The only thing worse than mistakes though are mistakes that go uncorrected. Unfortunately the first step to correcting a mistake is to admit it exists.
Ego can also get in the way and take management off into more interesting (or brag-worthy) endeavours than what's best for the business. I've seen this happen countless times where teams deliver absolutely nothing while working on the latest in tech. If the same ego-driven management style reaches high enough into the organization, and the company is profitable enough to support it, it's easy for a team to get away with this blunder for years (which I've also seen).
Copy-cat or Cargo-cult management is probably the next most common excuse for not examining results. Often I've heard "google does x" or "y is an industry best practice" or even "that's not Agile" without any discussion of whether a particular practice makes sense for the team's own particular goals, scenario, or needs. Often these managers feel that they're adhering to "best practices" and so the actual results will necessarily be optimal and won't need to be examined. I'm definitely a proponent of agile methodologies in general, but saying whether or not something is "Agile" explains nothing about it's utility to the organization. There are no sacred practices that should be allowed to escape scrutiny. Any practice that can't be shown to be adding value should be canceled immediately.
It's common as well when a manager does manage to pay attention to results that his/her methods are extremely flawed. Probably the most infamous example of this is how managers used to count lines of code to judge the productivity of a developer. Counting the number of commits is also ridiculously flawed in the same way: they're missing what it means to be productive in an engineering context entirely. The least productive engineers are doing things like building needlessly elaborate architectures and suffering from NIH when they could just use a far better 3rd party library. Yes, you do need to be able to determine if a developer is productive or not, but that doesn't mean that that determination can be quantified. There will be some very important things that you need to measure that aren't quantifiable, so managers need to be comfortable with qualitative measurements.
In short, I think it's pretty common for engineering management to be actively harmful to their team's speed and their product's quality. Because uncorrected management mistakes impact entire teams, it's quite easy for a manager to have an overall net negative contribution to the organization. Of course it doesn't need to be like this, but a lot of the common expectations of management will really need to change first.