One fine day in the trenches, a junior engineer was tasked with optimizing a particular web service call that was way too slow (sometimes slower than 20 seconds!).  I had just figured out how to use a performance profiler for that particular platform, so I thought I’d offer to show him how to use it.  His response was that he was on a tight deadline and didn’t have time to learn to use the profiler, and that he was just going to tune the database calls to make them perform better.
Later in the year when the problem still wasn’t solved, I had a chance to take it on myself.  The profiler showed me that this service call was itself making around 20 web service calls to a second system that could obviously add up to 20 seconds, if that other web service took 1 second for each of its responses (and the logs showed that sometimes it did).  It turned out that almost all of those web service calls were redundant and completely unnecessary.  Simply removing them (which took about 1 day of effort) shaved 19 seconds off this web service call’s duration in production in some cases.
The total execution time spent on database calls was 5 milliseconds.  A 50% improvement there would probably take a week’s worth of effort, and really only get you 2 or 3 milliseconds of performance improvement.
The moral of the story is that there’s a pretty simple process for optimizing a system:
(1) Identify and isolate the component that is tying up the most time in the system  (let’s call it the bottleneck)
(2) Optimize that component to make it less wasteful
(3) Repeat
Skipping step 1 isn’t a shortcut.  It’s absolutely critical to know that you’re optimizing the bottleneck, and not some other random component.
This is pretty obvious stuff for most experienced developers of course.  And while they seem to know to apply this to the systems that they work on, it’s more rare that they notice it outside of the code and see it in the process of software engineering in general.
The software development /process/ is a system as well, made up of components.  Requirements come in at one end, and deliverables go out the other end.  As people who are involved in that process, we naturally want it to be more and more efficient.  We are constantly trying to figure out how to optimize that system so we can deliver better software in less time.
So step 1:  Identify and isolate the component that is tying up the most time in the system.
Not every team develops software the same way, so this component of the process *could* be different team by team.  In practice though, I’ve always seen teams spending the most time doing these sorts of things:
* Refactoring poorly written code so that features can be added
* Learning/relearning existing code that has long since swapped out of all developers’ “mind pages” (for extension or bug-squashing).
* Debugging
* Performing Manual User Acceptance Testing (by both developers and testers)
* issue tracking (administration of that tool and issue in it)
* discovery and documentation of steps to reproduce
* triage (what bugs to squash first?)
* communication between customers, support, QA and developers
* customer support
* customer dissatisfaction leading to churn and reputation problems (affecting the entire organization)
All of these time consumers are affected directly or indirectly by our ability to control quality.  In my experience, the majority of the software development process isn’t hung up in requirements analyis, design, or implementation.  It’s hung up in all these Quality Control related issues.  Quality Control is the constraint that keeps us from creating the right software faster.  Steve McConnell confirms it in a more scientific way than I ever could:
The rest of us would do well to learn from a discovery made by IBM in the 1970s: Products with the lowest defect counts also have the shortest schedules (Jones 1991). Many organizations currently develop software with defect levels that give them longer schedules than necessary. After surveying about 4000 software projects, Capers Jones reported that poor quality was one of the most common reasons for schedule overruns (1994). He also reported that poor quality is implicated in close to half of all canceled projects. A Software Engineering Institute survey found that more than 60 percent of organizations assessed suffered from inadequate quality assurance (Kitson and Masters 1993). On the curve in Figure 1, those organizations are to the left of the 95-percent-removal line.
(http://www.stevemcconnell.com/articles/art04.htm — The whole article is extremely illuminating)
What this means is that if we want to move quickly, we need to take a good hard look at how we manage quality.  Of course this is counter-intuitive, and its actually hard to swallow, because ensuring quality can be mundane and tedious if you’re not doing it right.  But as developers we’d be silly to ignore the real bottleneck in the process, and optimizing for that.  Micro-optimizations elsewhere in the process will yield very little benefit compared to improving measures that ensure quality.  As developers, we love to argue about and discuss these micro-optimizations all day long:
* what text editor you use ( http://en.wikipedia.org/wiki/Editor_war )
* how important input speed is (http://steve-yegge.blogspot.com/2008/09/programmings-dirtiest-little-secret.html and http://imprompt.us/2009/code-fast/ ).
* IDEs vs. editors ( http://stackoverflow.com/questions/136056/ide-or-text-editor)
It sure seems silly though to be pursuing micro-optimizations, at the expense of tackling the real #1 time-suck:  Quality.

One fine day in the trenches, a junior engineer was tasked with optimizing a particular web service call that was way too slow (sometimes slower than 20 seconds!).  I had just figured out how to use a performance profiler for that particular platform, so I thought I’d offer to show him how to use it.  His response was that he was on a tight deadline and didn’t have time to learn to use the profiler, and that he was just going to tune the database calls to make them perform better.

Later in the year when the problem still wasn’t solved, I had a chance to take it on myself.  The profiler showed me that this service call was itself making around 20 web service calls to a second system that could obviously add up to 20 seconds, if that other web service took 1 second for each of its responses (and the logs showed that sometimes it did).  It turned out that almost all of those web service calls were redundant and completely unnecessary.  Simply removing them (which took about 1 day of effort) shaved 19 seconds off this web service call’s duration in production in some cases.

The total execution time spent on database calls was 5 milliseconds.  A 50% improvement there would probably take a week’s worth of effort, and really only get you 2 or 3 milliseconds of performance improvement.

The moral of the story is that there’s a pretty simple process for optimizing a system:

  1. Identify and isolate the component that is tying up the most time in the system  (let’s call it the bottleneck)
  2. Optimize that component to make it less wasteful
  3. Repeat

Skipping step 1 isn’t a shortcut.  It’s absolutely critical to know that you’re optimizing the bottleneck, and not some other random component.

This is pretty obvious stuff for most experienced developers of course.  And while they seem to know to apply this to the systems that they work on, it’s more rare that they notice it outside of the code and see it in the process of software engineering in general.

The software development process is a system as well, made up of components.  Requirements come in at one end, and deliverables go out the other end.  As people who are involved in that process, we naturally want it to be more and more efficient.  We are constantly trying to figure out how to optimize that system so we can deliver better software in less time.

So step 1:  Identify and isolate the component that is tying up the most time in the system.

Not every team develops software the same way, so this component of the process could be different team by team.  In practice though, I’ve always seen teams spending the most time doing these sorts of things:
  • Refactoring poorly written code so that features can be added
  • Learning/relearning existing code that has long since swapped out of all developers’ “mind pages” (for extension or bug-squashing).
  • Debugging
  • Performing Manual User Acceptance Testing (by both developers and testers)
  • issue tracking (administration of that tool and issue in it)
  • discovery and documentation of steps to reproduce
  • triage (what bugs to squash first?)
  • communication between customers, support, QA and developers
  • customer support
  • customer dissatisfaction leading to churn and reputation problems (affecting the entire organization)

All of these time consumers are affected directly or indirectly by our ability to control quality.  In my experience, the majority of the software development process isn’t hung up in requirements analyis, design, or implementation.  It’s hung up in all these Quality Control related issues.  Quality Control is the constraint that keeps us from creating the right software faster.  Steve McConnell confirms it in a more scientific way than I ever could:

The rest of us would do well to learn from a discovery made by IBM in the 1970s: Products with the lowest defect counts also have the shortest schedules (Jones 1991). Many organizations currently develop software with defect levels that give them longer schedules than necessary. After surveying about 4000 software projects, Capers Jones reported that poor quality was one of the most common reasons for schedule overruns (1994). He also reported that poor quality is implicated in close to half of all canceled projects. A Software Engineering Institute survey found that more than 60 percent of organizations assessed suffered from inadequate quality assurance (Kitson and Masters 1993). On the curve in Figure 1, those organizations are to the left of the 95-percent-removal line.

(http://www.stevemcconnell.com/articles/art04.htm — The whole article is extremely illuminating)

What this means is that if we really want to move quickly, we need to take a good hard look at how we manage quality.  Of course this is counter-intuitive, and its actually hard to swallow, because ensuring quality can be mundane and tedious if you’re not doing it right.  But as developers we’d be silly to ignore the real bottleneck in the process, and optimizing for that.  Micro-optimizations elsewhere in the process will yield very little benefit compared to improving measures that ensure quality.

As developers, we love to argue about and discuss these micro-optimizations all day long:

It sure seems silly though to be pursuing micro-optimizations, at the expense of tackling the real #1 time-suck:  Quality.

Learning new languages always makes me think about programming in a new way.  The different design decisions from the implementation of one language to those of another language always say something to me about the way the designer thinks about programming.  Learning python for instance has really made me reconsider the value of access modifiers (private / public).

Python simply doesn’t have access modifiers and that’s a design decision based on python’s adage that “we’re all consenting adults“. It’s not that pythonistas don’t care about encapsulation though: in python, the convention (that is not enforced in any way whatsoever) is to denote a private method or property by prefixing its name with an underscore. People that are used to languages with access modifiers (c#, java, actionscript) will be recoiling in horror at this point, but let’s take a look at the trade-offs.

What do access modifiers get you that a convention doesn’t?

Clearly from the consumer of the class’ point-of-view, access modifiers just remove functionality that would otherwise exist.  Sure you’re not supposed to use that functionality, but make no mistake about it, functionality is revoked.  There is no gain whatsoever for the consumer.

From the perspective of the author of the class, access modifiers enforce your public interface so that even if the consumer wants to shoot himself in the foot, he can’t.  There’s really no advantage to the author though: the author just does this to stop the consumer from relying on some internal that might change in the future.  Isn’t just marking something as internal (with an underscore prefix perhaps)  enough though?  If that’s the convention of the language (and the programmers involved know the language) , then hasn’t the author absolved himself of responsibility?

Now which side you fall on in this argument is a matter of politics I think.  You either think developers are competent enough to make responsible decisions, or you think they need to be babysat.  Personally I think that developers that are always babysat will continue to need to be babysat in other areas as well.

So now that we all agree that access modifiers are unnecessary (right? right?), here’s why I like not having them in python:  Better unit-testing for two reasons:

(1) I get to unit-test internals.  Yeah, I know that every branch of every internal method should be reachable through the public interface and I know that it makes my tests brittle to future change, but sometimes I still like testing a particularly complex internal methods to make sure they’re as solid as I want them to be.  It’s a much faster feedback loop to be testing this way than to always try to reach that internal method through the proper public interface.  Maybe that type of test is throw-away at some point in time, but it sure is useful while doing TDD of tricky internal methods.

(2) Free Dependency Injection!  If my class-under-test has some dependency instantiated to a ‘private’ property, my tests can swap a mock into its place at some point.  Take a look at how DI is done in other languages/environments to see just how comparatively simple and useful this can be.

The bottom line for me is that access modifiers don’t empower me in any way.  If it wasn’t for my exposure to a new language, I wouldn’t have been able to re-examine what I previously had considered a universal truth.

At work we’ve been doing a bunch of technical interviews for new hires, and I thought I’d give the Fizzbuzz test a go for evaluating candidates.  I realize that it’s a famous interview test, but in practice none of the candidates have heard of it before.  I think that if someone had heard of it before, then that would be okay too, because at least they’ve demonstrated that they’re reading blogs and keeping abreast of current trends in their craft.  We throw a curve-ball into it anyway.

First the interesting parts!

Almost no one knows the modulus operator

I can’t think of an algorithm for fizzBuzz off the top of my head that doesn’t either use the modulus (%) operator, or make use of similar check-the-remainder functionality that is probably better done with the modulus operator.  We let the candidate write the program in any language they want and they all have chosen a language with a modulus operator, so I’m surprised that around 2/3rds of the candidates don’t know it and have to improvise with a function they make up themselves.   Some of them even seem to be good programmers otherwise.

Many still fail to implement their program properly in the language of their choice

A number of candidates picked Java for their implementation language, and then proceeded to write a function.  Java doesn’t have functions;  it has methods, so you better be ready to wrap that method with a class.  This might seem like a nitpick, but we didn’t pick the language, the candidate did.  Java certainly wouldn’t have been my first choice for a problem of this level of simplicity.

Many candidates have a spectacular number of basic syntax errors

I’m not expecting people to be the human equivalent of a syntax-checker, but let’s face it… FizzBuzz can be done in around a dozen nice lines of JavaScript:

function fizzbuzz(){
  out = '';
  for (i = 1; i <= 100; i++){
    switch(true){
      case (i % 3 == 0 && i % 5 == 0): out += "fizzbuzz"; break;
      case (i % 3 == 0): out += "fizz"; break;
      case (i % 5 == 0): out += "buzz"; break;
      default: out += i;
    }
  }
  return out;
}


If you have 4 or 5 syntax errors in that much code, there’s got to be something inherently wrong.

Nobody unit-tests — even the ones that say they do
Here’s the spin we added to it:  we request that the candidate write a unit test for it afterward.  It doesn’t really matter to me if they know a unit-testing framework (though I think it says something about their testing experience right away if they don’t) as long as they get the basics of an adequate test expressed.  I tell them they can reduce the requirement of 100 iterations down to 20, and they can change their program however they want to make it more easily testable.

On the positive side, the people with the System.out.println() statements end up changing those in favour of returning the string.  I’m happy to see that most are capable of seeing that necessity.  I think it’s a good confirmation of their ability to solve problems in general.

On the negative side, no one actually knows how to use any particular unit-testing framework.  Many profess to have used JUnit, but few are actually using it even remotely correctly in the code they write.  We’re also looking for a test that basically compares some example string to the output of the function like this:

function testFizzBuzz(){
  assertEquals("12fizz4buzzfizz78fizzbuzz...", fizzbuzz());
}


Unfortunately everyone seems to want to repeat all the same logic as the code-under-test in the test itself.  This is a serious unit-testing no-no.  If you use the same code in the test, how do you know that that code is right?!?

A coding exercise is absolutely vital to any technical interview

Fizzbuzz is really invaluable to us in actually assessing a person’s ability to code.  I’ve realized that a resume really gives no indication of that whatsoever.

I also realize that it’s meant to be ridiculously easy and doesn’t really show how they’ll do on hard problems, but you need it to be ridiculously easy so that it’s possible to complete it within a half hour or so.  Combine it with the unit-test writing portion, plus any discussion you want to have about it and it will probably take around 40 minutes (I realize that people get nervous too, writing code to be ripped apart, and that a whiteboard isn’t anything like the computer that we actually work on, so we try to take that into consideration.).

That’s enough time to tell who has basic coding skills, and who has basic unit-testing skills.  You’d be surprised how many don’t.  I simply can’t imagine ever doing another technical interview that doesn’t involve some coding exercise.

Unless you’ve done a fair amount of unit-testing, it’s simply impossible to understand all the advantages that are to be gained from it.  A lot of the focus has naturally been on the positive effects of unit-testing on the current state of the code:  it helps you find the bugs.  Duh.  But there are also huge benefits to the future state of that codebase that are often overlooked.  In a nutshell, unit-tests ensure the ongoing health of your codebase and the consistent flow of your development.  Here’s how:

Better regression testing when the code changes.

Manual testing is boring, tedious, time-consuming, and above all, error-prone.  If you invest the extra time to automate a test, and then the machine can do the tests for you in a way that’s superfast, complete, and mechanically consistent, you’ll test more often and more comprehensively with automated tests.

Better on-boarding.

Unit-tests serve as a kind of indicator of how code should be consumed, and what it is expected to do.  They’re not meant to supplant good (internal or external) documentation, but unit tests can serve a lot of the same goals as documentation and they tend to be better maintained;  if you don’t maintain your tests as code changes, you’ll get failed tests.  Documentation doesn’t have an automatic check like this, and can go stale more easily without proper vigilance.  New-comers to your codebase will be able to make changes with a great deal more confidence too, because there’s an easy way for them to tell if their changes have caused a regression.

Improved velocity.

Code written to be unit-tested is code that’s better factored.  I strongly believe that without significant unit-testing experience, a developer can’t imagine the myriad of ways that he/she factors things improperly.  It certainly was an eye-opener for me.  Better factored code leads to faster development (easier code re-use, better understandability, etc).  Couple that with faster testing and fewer defects and you get vastly improved velocity in your development process.

It’s natural for managers and developers to question the extra time taken to write unit-tests, and to put value on them only equal to the number of defects they uncover when first written, but there’s a whole lot of extra value that you get from them afterward as well.

This might be one of those things that has to be experienced to be understood.  It definitely was for me.