caines.ca

Agentic Bug Fixing: /undefect

2026-06-17T00:00:00Z

I've got a skill that drastically improves code quality and architecture and it's powered by defects.

I'm sure you've already had many scenarios where you've had your agent fix the same bug 3-4 times. That's not just a defect, it's a systemic failure.

Sure, we want to fix the bug -- but it's even better to fix the problem that caused the problem: to learn what went wrong and what conditions made us bug-prone. Fixing the bug isn't as powerful as fixing the conditions that created it.

That's what /undefect is for: it investigates a defect deeply and makes it easy to detect -- without fixing it yet. (If you'd rather see it work before reading the design, skip to what a run looks like.)

You can grab the skill from my skillz repo on GitHub, or you can click here to expand it here.

---
name: undefect
description: >-
  Investigate a software defect without fixing it. Use when the user wants a
  complete bug investigation, root-cause explanation, early detection measures
  through types/tests/logging/user feedback, verification that each detection
  measure catches the defect, architectural recommendations, and a scan for
  similar defects from similar causes, while explicitly preserving the defect
  for a later fix.
---

# Undefect

Use this skill to understand a defect deeply and make it easy to detect, without fixing the defect yet.

## Prime Directive

Do not fix the defect as the outcome of this skill.

If investigation requires temporarily fixing the defect to prove causality, do it only as an experiment. Before finishing, revert the temporary fix and leave only detection, observability, documentation, or user-feedback changes that do not correct the defect itself.

Our main goal is to either eliminate the potential for this class of defect, or failing that, make detection simpler and faster.

## Workflow

1. Establish the defect precisely.
   - Reproduce it from the user-visible behavior, a failing command, a failing test, logs, or a minimal code path.
   - Record the expected behavior, actual behavior, trigger conditions, affected surface area, and any non-reproduction cases discovered.
   - Prefer local deterministic reproduction before production reproduction. Avoid production mutation unless the user explicitly allows it.

2. Explain why the bug exists.
   - Trace the code path from input/event to wrong outcome.
   - Identify the missing, incorrect, or duplicated responsibility that permits the defect.
   - Before calling it a local bug, ask whether the failing behavior is caused by two sources of truth, duplicated policy, split ownership, or drift between surfaces that should share one contract.
   - If the defect involves ordering, prioritization, authorization, validation, naming, state transitions, derived status, or other policy, explicitly identify the canonical owner that should decide that policy. If no clear owner exists, call that out as part of the root cause.
   - Distinguish facts from inferences.
   - If useful, prove causality with a temporary experiment, then revert that experiment.

3. Add the fastest local detection mechanism that can reveal the defect without fixing it.
   - Prefer compile-time/type-checking when realistic.
   - Otherwise prefer a focused unit/integration test.
   - Use end-to-end tests only when lower-level tests cannot express the failure.
   - Prefer detectors at the contract boundary or canonical policy owner over detectors in only one consumer. For drift bugs, add a detector that compares sibling surfaces or forces them through the same shared contract.
   - Keep the detector narrowly scoped to the defect.
   - Run the detector and confirm it fails or signals because the defect still exists.

4. Add the fastest production/system feedback mechanism that can reveal the defect without fixing it.
   - Prefer structured logging for invisible backend failures or missing side effects.
   - Add user feedback when the user can recover, retry, or needs to know work did not happen.
   - Avoid redundant detectors. If a type check makes a unit test unnecessary, or a test makes extra logging unnecessary, skip the redundant measure.
   - After each measure, trigger the defect and confirm the measure fires.

5. Preserve silent-failure prevention.
   - The final state should not allow the defect to remain completely silent for both developers and users.
   - Aim for one fast local signal and one fast production/user-facing signal.

6. Look for enabling architecture problems.
   - Identify refactorings that would make this class of defect harder to create.
   - Favor structural improvements: single source of truth, typed contracts, centralized side-effect helpers, parameterized APIs, exhaustive switches, shared adapters, or stronger invariants.
   - When there are multiple implementations of the same business policy, recommend combining them behind one owner instead of patching each caller independently.
   - When behavior differs by surface (UI, CLI, API, worker, integration), decide whether the difference is intentional. If it is not intentional, recommend a shared contract and list every surface that should consume it.
   - Present these as recommendations unless the user asks to implement them.

7. Look for similar defects from similar causes.
   - Search for nearby code paths, sibling features, duplicate implementations, adapter variants, route variants, and test gaps that share the same missing, incorrect, or duplicated responsibility.
   - Use the root cause as the search key, not just the symptom text.
   - For each plausible similar defect, report whether it is confirmed, ruled out, or still uncertain.
   - Add detection for confirmed similar defects only when it fits the user's requested scope and does not fix the defect.
   - Do not broaden into unrelated cleanup; keep the pass focused on defects enabled by the same cause.

## Verification Rules

After every detection or feedback measure:

- Trigger the defect again.
- Report the command, test, log line, or UI feedback that proves the detector worked.
- If the detector does not fire, improve it or explain why that detection route is not viable.

Before finishing:

- Re-run the fastest detector.
- Confirm the defect is still unfixed unless the user explicitly changed the goal.
- Confirm any temporary fix experiments were reverted.

## Final Response Shape

Lead with the root cause and the detection measures added.

Include:

- Why the defect exists.
- What detection/feedback was added.
- How each detector was verified.
- Whether the defect remains intentionally unfixed.
- Any architectural refactorings recommended for a later change.
- Similar defects from similar causes that were confirmed, ruled out, or left uncertain.
- Any tests or commands that were not run.

What a run looks like

Here's a (lightly fictionalized) run on a drift bug -- the case /undefect is most interesting on -- as it actually scrolls by in the terminal:

> /undefect customers see "free shipping unlocked" at a $50 cart subtotal,
  but checkout still charges them $6.99 shipping

● I'll investigate without fixing it: reproduce, find the root cause, and
  leave behind detection that fails *because* the bug is still live.

● Search(pattern: "free.?shipping|shippingCost|FREE_SHIPPING")
  ⎿  6 matches across 4 files

● Read(cart/summary.js)
  ⎿  read 24 lines
● Read(checkout/shipping.js)
  ⎿  read 18 lines

● Reproduced. The threshold lives in two places and they disagree at the
  boundary:

      cart/summary.js       subtotal >= 50   -> banner says "free"
      checkout/shipping.js   subtotal > 50    -> charges $6.99

  At exactly $50 the cart says free and checkout charges. This isn't a local
  bug -- it's two sources of truth for one pricing policy, with no canonical
  owner. Both surfaces re-derive the rule, so they're free to drift.

● Adding a local detector at the contract seam -- it forces both surfaces to
  agree on the same input instead of testing just one consumer.

● Write(shipping-policy.contract.test.js)
  ⎿  wrote 6 lines

● Bash(npx vitest run shipping-policy.contract.test.js)
  ⎿   FAIL  shipping-policy.contract.test.js > banner and checkout agree at 50
          expected false to be true
          - Expected:  true   (banner: free)
          + Received:  false  (checkout: charged $6.99)
        Tests  1 failed | 5 passed (6)

● Good -- red, and only at 50. The reproduction is now nailed to a test.

● Adding a production signal so the silent version of this stays loud: a
  structured log when the two surfaces disagree at request time.

● Update(checkout/shipping.js)
  ⎿  +3 lines (detection only -- does NOT change the charge)

● Bash(curl -s localhost:3000/checkout -d subtotal=50 >/dev/null; tail -1 dev.log)
  ⎿  WARN shipping_policy_drift {"subtotal":50,"banner":true}

● Fires. The $50 cart is still charged $6.99 -- I have not fixed the bug.

● Sweeping for the same *cause* ("threshold re-derived per surface"), not the
  symptom text:
  ⎿  email/receipt.js             CONFIRMED  hardcodes `subtotal > 50`
     promo/eligibility-worker.js  CONFIRMED  uses `>= 50` AND a stale `45` flag
     mobile/CartScreen.tsx        UNCERTAIN  calls /cart/summary -- needs a trace
     admin/order-export.csv       RULED OUT  reads persisted shipping_charged

● Done. The defect is still live, and now it's cornered:

  Root cause   Two sources of truth for the free-shipping threshold, no
               canonical owner. `>=` vs `>` drift breaks the $50 boundary.
  Detection    1 failing contract test (red at $50) + 1 drift log line.
  Verified     Watched the test fail for the right reason; saw the log fire.
  Still broken Yes -- $50 carts are still charged. Fix intentionally deferred.
  Recommend    One owner: shippingPolicy.qualifiesForFree(subtotal), called by
               both surfaces. The contract test then guards the seam forever.
  Similar      2 confirmed, 1 uncertain, 1 ruled out (above).

One $50 bug, cornered by a test I watched go red -- and three more places the same policy is duplicated, surfaced by the sweep.

Here's the thinking behind the design.

Why forbid the fix?

Like I said, this skill purposely leaves the defect unfixed. That's so you can check whether the tests or other measures actually find the defect. I can't tell you how many times I've seen people write tests after the fix, and they get coverage but don't actually ensure the defect can't happen again. You have to write your tests first and see them fail in order to "test your tests".

The moment a bug is fixed, the agent loses its reproduction. There's nothing left to point a new test at, nothing to confirm a log line fires for, nothing to prove an architectural change actually closes the hole. The defect is the only ground truth you have, and a fix destroys it.

So /undefect keeps the bug alive on purpose, and uses it as a target to verify against. Every detection measure it adds gets run against the live defect to prove it actually fires. By the time the skill finishes, you have a failing test (or a type error, or a log line) that you've watched fail for the right reason.

Root cause, not symptom

The symptom above was "checkout charged $6.99 at a $50 cart." The cause was "we handle pricing differently in two different places instead of having a single clear source of truth."

Once you've got the root cause, you've identified a pattern that allows you to ask:

Are there other possible defects caused by this pattern?
Do we have any safety measures in place to help eliminate/prevent/detect this pattern in the future?

You want to be able to inoculate both cross-codebase and into the future. Every defect is an opportunity to improve the system that the agent is working in.

One fast signal locally, one in production

Steps 3-5 are a deliberate pair. A test that catches the bug on your laptop does nothing for the version of this bug that happens silently in production six weeks from now. A log line does nothing for the version a teammate reintroduces before they ever deploy. So the skill aims for both: one fast local detector and one fast production/user-facing signal, and explicitly tries to avoid the defect ever being completely silent again.

It also tries not to pile on redundant detectors. If a type makes the bug impossible to express, you don't also need a unit test for it. The cheapest, fastest signal that actually fires wins.

Verify everything fires

I don't trust a detector I haven't watched fail. Neither should the agent. The verification rules force it to trigger the defect again after each measure and report the exact command, test, or log line that proved the detector worked. This is the same instinct as TDD's red step; a test you've never seen fail is a test you can't trust.

LLMs are especially prone to writing a confident-looking test that passes for the wrong reason. Making the defect prove the detector closes that gap.

The similar-defects sweep

Step 7 is the payoff for doing root-cause analysis properly: The same missing validation probably exists in the sibling route. The same drift probably exists on the CLI that the UI just exposed. So the skill goes looking, and reports each candidate as confirmed, ruled out, or uncertain.

This is where one bug investigation quietly prevents three more.

Why this is a separate skill from fixing

You could imagine folding all of this into a normal "fix the bug" flow. I keep it separate on purpose. Fixing is goal-directed and wants to converge fast; investigation wants to stay open and suspicious. Mixing them means the convergence pressure of "make it green" wins, and the investigation gets skipped. By making /undefect a thing you run before you fix -- and one that refuses to fix -- the investigation actually happens.

Better Taste == Better Autonomy

Concretely, this is what changes for me day to day: I can hand the agent a list of 20 defects, tell it to run /undefect on each, implement the recommendations, and fix the bugs -- and trust that what comes back is investigated, not just patched.

/undefect is really just my taste in debugging, written down -- the same way /defrag is my taste in refactoring. I highly doubt that agent autonomy will be sufficiently achieved through bigger/better models alone. We need to inject our human taste into the recipe or be ever-present babysitters. Every skill like this is one more thing you no longer have to be in the room for.

Agentic Refactoring: /defrag

2026-03-10T00:00:00Z

Here's a question: if you were told you were going to get the biggest, worst codebase that you had to work on, sight-unseen, but you could magically apply one refactoring to it, what would that refactoring be?

good naming?
no god classes?
tabs instead of spaces? 🙃

I think it's a good question, because:

The magic largely exists all of a sudden in the way of LLMs.
I want to be reading less agentic code, but I don't want slop
LLMs don't handle slop well either

Slop is the reason that you can ask an agent to build out a full 3d game, but it explodes into a fine mist if you ask it immediately after to change the character's shirt color. LLMs need good factoring / code organization just like humans do.

To that end I've been running an agentic refactor skill for Claude Code that I call /defrag. I run it after a large new feature or after a couple of smaller features to keep the code maintainable and to keep the agents effective. It also happens to contain my 14 answers to the original question.

---
name: defrag
description: Analyze code for refactoring opportunities and suggest the top 10 highest-value improvements. Use when the user says /defrag, "refactor", "clean up code", or asks for code improvements.
---

# Defrag - Code Refactoring Analysis

Analyze the specified file(s) or the user's current selection and identify the top 10 highest-value refactoring opportunities.

## Refactoring Types to Look For

1. **shorten_file** - Files >300 lines should be broken into multiple coherent files
2. **shorten_function** - Functions >50 lines should be broken into smaller functions with descriptive names
3. **reduce_nesting** - Replace nested conditionals with guard clauses and early returns
4. **extract_function** - Extract repeated or complex code into well-named functions
5. **rename_for_clarity** - Improve variable/function names that are unclear or too short
6. **simplify_conditionals** - Simplify if/else logic, use early return where possible
7. **extract_constants** - Move magic numbers/strings to local constants (NOT global constants files)
8. **consolidate_duplicates** - Merge duplicate code blocks
9. **modernize_syntax** - Use modern language features (destructuring, optional chaining, etc.)
10. **avoid_globality** - Move global items closer to where they're used; no utils/constants/managers files
11. **optimize_imports** - Clean up unused or poorly organized imports
12. **remove_dead_code** - Remove unused code.  In typescript you can use knip for that.
13. **add_tests** - Identify code lacking test coverage
14. **break_up_hotspots** - look at the last 30 days of git history for the 5 files with the most changes.  Suggest decomposition for any file that's changed more than 30 times.

## Scoring (Value Score 0-100)

Calculate each refactoring's value using these weights:
- **Readability improvement** (35%)
- **Maintainability improvement** (30%)
- **Bug risk reduction** (25%)
- **Performance impact** (5%)
- **Scope size** (5%)

## Output Format

For each refactoring opportunity, provide:

### {rank}. **{type}** (Value: {score}/100)
- **File:** `{filepath}:{start_line}-{end_line}`
- **Description:** {one-line description}
- **Rationale:** {why this helps}

**Before:**
```{language}
{current code snippet}
```

**After:**
```{language}
{refactored code snippet}
```

---

## Instructions

1. Read the file(s) the user specifies (or use their IDE selection)
2. Analyze for all refactoring types listed above
3. Score each opportunity using the weighted formula
4. Sort by value score (highest first)
5. Present the top 10 refactorings
6. End with: "Found {N} refactoring opportunities. Top 10 shown with average score: {avg}"

## Important Rules

- **Do NOT apply changes** - This is analysis only (plan mode)
- **Sort by value_score** - Highest value first
- **Show before/after code** - For each suggestion
- **Be specific** - Include exact file paths and line numbers
- **Follow YAGNI** - Don't suggest over-engineering or future-proofing
- **No global constants files** - Constants belong near their usage

After presenting, ask: "Would you like me to apply any of these? Say 'apply #1' or 'apply all'."

Here's the thinking behind some of these rules:

About function length and file size

I got the ideal numbers by asking the LLM for ideal maximums. It probably doesn't know any better than I do, so feel free to fill in your own choices, but if you're coding agentically for long you'll hit whatever maximum you set eventually. Why this helps: These maximums are two primary drivers of abstraction. Just like a human, the LLM can look at a file name or function name and assume what it does from the name instead of having to read all the code in it. That means it has to read a lot less code, and it uses a lot less context.

I also asked the LLM "which is more expensive for LLMs: long files or long functions?" It told me that it can grep through long files but it has to read the entirety of a function, so long functions are more expensive for it. I don't actually totally believe the LLMs with these answers. I'll need to run some evals on all of this at some point, but I think it's a good starting point.

Reducing code

I feel like reducing code has got to be one of the easiest ways for a codebase to require less context. Larger codebases are usually harder for an llm to manage than smaller ones with all else equal. To that end, I try to find and remove dead code and deduplicate code. This didn't require more proof for me anyway.

Reducing complexity

LLMs are better at complexity than us, but they have all the same issues in following code that we do. For that reason I try to reduce nesting, early return when possible, and avoid making things more global than they need to be.

Adding Tests

Tests are a no-brainer part of good harness engineering.

Breaking up Hotspots

I call files that are changed really frequently "hotspots". The problem with hotspots is that if you're using many agents in parallel, they easily cause merge conflicts. It's just another good signal that a file needs to be broken up.

What I removed: 3rd party library suggestions

I used to have a rule that asked it to suggest a 3rd party library for any repeated pattern it finds. I removed it after an LLM told me that 3rd party libraries can be a liability if it's not trained on them. It's often easier to solve patterns internally where the source code is available to read.

A few interesting things I've seen with this approach

It actually works pretty well. If you're refactoring with this often, it will only find issues in code you've worked on recently and haven't refactored. So it's really finding and fixing things as it goes, if you keep the discipline to refactor frequently. I have a codebase that's approaching 250K LOC (that I've never looked at) that is still workable for agents.

This approach is a little outside of regular harness engineering where we'd usually be trying to make sure that the LLM produces the code that we want. The approach is after-the-fact, and can be applied to existing terrible codebases. I think that's useful because even if all your additions to the codebase are excellent quality, the codebase can still drift into something unmaintainable.

Ultimately none of this is all that scientific which irks me a bit. I tried to ask the LLM about things that made sense and incorporate its guidance, but I think running some real evals (hitting it with a barrage of tests that try things and measure context-usage as well as results with different rules/tasks/codebase) will get more reliable results.

I do plan to do that eventually and post updates. In the meantime, I'd love to hear from you about how /defrag works for you or what other strategies you think I'm missing.

Context Management: Why your AI Keeps Forgetting What You Said

2026-02-21T00:00:00Z

One of the main reasons I see people getting poor results from agentic coding is that they're not managing their context properly. They're getting results like:

The AI forgets clear instructions you gave multiple times.
It keeps suggesting an idea you've rejected multiple times.
It gets desperate about completing the task at hand quickly rather than doing it well.
It assumes fundamental things that it should be asking about.

What is context?

The "context" is the LLM's short-term memory of:

any instructions / documentation that you pre-loaded (eg claude.md, skills, agents, etc)
your conversation with it
what it's done / read during the conversation

By "managing your context", I generally mean: you try to keep your context usage below ~60% where possible. The exact number will depend on the model used, but 60% seems to be a good rule of thumb. The poor results you get from high context usage are commonly called "context rot".

Why do you want to keep context usage low?

When Claude Code gets to 95% it will automatically "compact", meaning it will automatically create a summary of the current context and restart from that. The summary will have to remove details in order to be smaller and they may be details you care about. This is one way the LLM "forgets" your instructions, or "forgets" that it already tried a particular solution unsuccessfully.
LLMs pay more attention to the beginning and end of your context (ie the "lost in the middle" problem). The bigger the middle gets, the bigger the opportunity for it to "forget" vital information in the middle.
LLMs are trained to prioritize helpfulness, and as context degrades their reasoning, they seem to compensate by becoming more "agreeable"—asking fewer questions, making more assumptions, and skipping verification. In my experience this manifests as them acting more desperate and doing dumber / more reckless things.

Tools for managing context in Claude Code

You can check your context usage in Claude Code with the /context command anytime to see what it currently is. You can reset your context by clearing all memory of the current conversation with /clear.

You can also try to reduce your context usage by clearing it and restarting with a summary of the current conversation with /compact as well. It's basically the same "compaction" that happens automatically when you get to 95% context usage, but you control the timing.

Tactics

Clear aggressively

The main thing you'll want to do is to run /clear as often as you can without losing valuable conversation.

Check your starting context

You also need to check what your context usage is when you first start a conversation. You can call /clear and then /context immediately after to see what you're starting with. Hopefully you haven't overburdened Claude Code with a massive amount of documentation, MCP servers, or skills that have you starting out every conversation already pretty high. You may have thought "I'll give Claude all the information and power that I can to get the best results", but that can often backfire.

Use sub-agents

Using sub-agents can be a help as well. When you spawn a sub-agent, it gets its own fresh context window and reports back only its results, saving the main agent from all that context-pollution.

Manually compact

I personally like more control and transparency than /compact, but I like the idea. If I'm in a long valuable conversation, I like to have it create a summary in a temporary summary.md file that I can check for completeness and accuracy. If it looks good, I /clear and have it read that file in to continue from. This works really well if it's in the middle of long list of tasks.

Use plan mode

Use plan mode! It's great for conversationally iterating with Claude Code about what you want to do (both asking and answering questions, and sometimes asking for questions) until you have a well-conceived plan. Claude Code will then let you accept the plan, clear context and start execution fresh.

Set up context-tracking

Set up better context usage tracking. In Claude Code, I set up a custom status line that shows me the context at all times:

Create ~/.claude/statusline.sh that shows context % with color-coded warnings (requires jq to be installed):

#!/bin/bash
input=$(cat)
PCT=$(echo "$input" | jq -r '.context_window.used_percentage // 0' | cut -d. -f1)

if [ "$PCT" -ge 70 ]; then
    echo -e "\033[31m⚠️ CONTEXT ${PCT}%\033[0m"  # Red
elif [ "$PCT" -ge 50 ]; then
    echo -e "\033[33mContext ${PCT}%\033[0m"     # Yellow
else
    echo -e "\033[32mContext ${PCT}%\033[0m"     # Green
fi

Make it executable with chmod +x ~/.claude/statusline.sh, then add this to ~/.claude/settings.json:

{
  "statusLine": {
    "type": "command",
    "command": "~/.claude/statusline.sh"
  }
}

Watch for symptoms

Ultimately you need to watch for symptoms: If it's doing reckless things, ignoring instructions and generally giving poor results, it's probably time for a /compact or /clear. Don't let yourself get stuck in a doom loop where it makes the same mistakes over and over and you keep telling it to fix the same things.

Managing context is a skill. At first you'll need to consciously check your context usage and remind yourself to clear. Over time you'll start to feel when the context is getting bloated—you'll notice the quality degrading before you even look at the percentage. That's when you know you've internalized it.

The goal isn't to obsess over the number. It's to get consistently good results by giving the LLM the space it needs to think clearly.

Agentically Ambitious

2026-02-09T00:00:00Z

One thing I've noticed when working with others while coding with agents is that many people are still working at too low of a level by default.

Level 0: Line-by-line dictation

Obviously if you're telling it line by line what to type (and no one is -- this is a straw man), you're not getting the value out of the LLM that you could be.

Level 1: Technical solutions

The next level up is specifiying technical solutions in detail, eg "Create a model and a controller for retrieving rows from the widget table". This is a common level for many people that have been burnt by older less-effective models. It's a waste because the agent knows how to build out CRUD endpoints for any web framework under the sun. Or if it doesn't, it can just follow examples in your codebase. It's extremely good at this now.

Level 2: System properties

An improved level that I see people default to frequently is describing properties of their system like Create an api endpoint GET /api/widget/234 that gets widgets. This is definitely an improvement, but it's still not as high level as we can get: nothing changes in the world when this endpoint exists. Backends need frontends to create full products. The example I've given here can still be absolutely crushed by the agent in most cases. You may still be at too low of a level.

Level 3: Product-oriented

A further improvement level is when you're coding agentically across all the necessary services in a way that is user-facing or product-oriented. eg "Create an orders page to show orders" that is run on a mono-repo or in a parent folder holding all the necessary repos. More often you're describing real user value instead of parts of the system.

It's worth noting that this is the lowest level that non-programmers will be operating at, so if non-programmers are blowing you away in productivity, now you know why. You're being silly if you let that happen!

Level 4: Problem-first and Level 5: Exploratory

And there are more levels even further outside! Instead of giving it a spec, you can give it a problem to solve and have it write its own spec (There are lots of tools for this like openspec or spec-kit). Or instead of giving it a problem to solve, you could just ask "what should we do next?"

I generally pick the highest level of abstraction that I think it can pull off. That's going to get you the biggest bang-for-the-buck when it works. The one where we're just supplying product specifications works extremely well today if your codebase doesn't suck.

In the end, it may not always do what you want. When it does something you don't like, you can throw it away and restart or iterate with it to where you want to go. But these new frontier models are so much better than previous models that it's worth a shot at trying something at a higher level or even without so much detail. And sometimes it has a better idea than the one you were going to specify which for me is just pure magic.

I would strongly consider being more ambitious if you're working at one of the lower levels here. And if it fails at a higher level, think about how you could make it work (eg skills, prompts, agents, documentation, etc) if you were to give it another shot. I realize that with previous models you needed to specify the hell out of everything, but the technology has changed considerably in the last 3-4 months. Try dropping to a lower level only after you've proven that the higher level won't work.

I also think the exact technical details of the solution matters a lot less now that you can refactor to what you want cheaply and easily as well, whenever you want. As long as you have the functionality, the performance, the UX, etc that you need, the exact code isn't all that important. I know that's going to blow a lot of minds, but I do encourage you to consider it.

If you're one of those people that's been using the agent for everything, but you don't feel like you're able to get the productivity gains that other people are seeing it could be that still working too far down into the details. The more ambitious you are, the more agentic you are.

Vibe Reviewing

2026-01-26T00:00:00Z

Steve Yegge shipped 225,000 lines of Go code he's never read. I just hit 170k lines on a side project without looking at the code. Vibe-coding is old news at this point, but what about vibe-reviewing? And possibly more terrifying: Can we build software without reading the code at all?

In the vibe-coding book, Yegge describes reviewing 10,000 lines of code a day. But he's not reading for elegance or comprehension. He's reviewing outcomes -- test results, whether the thing works. Code review becomes behavior review.

I'm exploring the same territory.

The Problem with Human Review

Sure, human code reviewers are slow and unreliable for stopping defects. The invention of pull-requests didn't bring in an unprecedented new age of software quality.

But the real problem with code review as a quality gate is that it comes at the end of the development process.

Code reviews as a quality gate are a massive source of waste where we discover problems way too late, requiring fixes that are more expensive than they need to be. My favourite W. Edwards Deming quote:

Let's make toast the American Way: I'll burn it, you scrape it.

If you've ever had a PR where someone says "I have an idea for a better approach entirely", you're scraping toast. Couldn't we instead just improve how we make things from the start and eliminate as much "scraping" as possible?

Even now in the age of AI, there are a million agents to automate PRs instead of employing agents to eliminate the need for review entirely. I've rarely heard anyone talk about using the same agents in precommit as they do for pull requests.

We've Done This Before

Remember when we had human QA gates before releases / deployment? Someone had to manually approve that the software worked before it could go out. Then we built automated tests and CI/CD pipelines and staged roll-outs and feature flags, and good observability and we managed to remove the humans from the loop. It was hard! We thought releases were so important that only humans could approve them. But then we figured out a better way.

The same pattern can happen with code review. We currently have human gates before code merges. But if we can automate the review -- really automate it, not just run some linter bot that everyone ignores -- we can remove humans from that loop too.

AI doesn't have to be perfect. It just has to be better than what we're doing today. (There's no question that it'll be faster. Waiting for a code review has got to be in the top 3 of software development soul-crushers.)

"But Review is for Knowledge Sharing!"

Common objection: code review spreads context across the team. Everyone learns the codebase by reviewing each other's code.

But most of that context is never used. You learn things in review that you never actually need. And deep codebase knowledge is devalued now anyway -- AI can navigate it for you.

Why learn preemptively when you can ask just-in-time? When you actually need to understand something, ask an agent. The context is there when you need it. You don't have to carry it around in your head.

Fresh Eyes Without Reading

I wrote myself a couple of useful Claude skills for review of the agent's work without looking at the agent's work.

The first is obviously /review . You can jam every opinion and quality check in there that you want. This is what you'll iterate on: every time you find an issue in a code review or a defect, consider adding the check to /review . Soon it will have a barrage of checks in it so extensive that no human can compete.

The second is /summary . All it does is summarize the code changes. It's useful for me to run on a clear context to see if the AI can figure out what the code is trying to do. And so it can find things that it's doing that are not coherent with the rest of the effort.

I use them both on a clear context ( /clear in claude code terminal ) so they have no hints about what to expect, and they have no biased agent (motivated to land the PR) interpreting the output. I don't read the code, I just point a fresh agent with these skills at it and see what comes back.

Lastly, things can still happen over time that make the code suck. Files get long, functions get long, "single sources of truth" show up in multiple places. I have a skill called /defrag that helps me find high-value refactorings. Basically everything that improves context management and quick semantic understanding.

ensures short files/functions
reduces hotspots (files that change all the time)
consolidates sources of truth
improves naming
removes dead code
adds missing tests.

The agents need refactoring just like we humans do if they're going to effectively and quickly navigate the codebase. This is the primary way I keep the agent successful as the codebase gets large. If it has to use up all its context figuring out your codebase, it will have no context left to intelligently handle your prompts.

I also ended up building a PR tool for myself for handling code review without reading code:

You still need to understand the code (at this point!) but you don't need to pore through it to figure out what needs to be fixed. AI can give you understanding at whatever resolution you need. Maybe you don't need to read all the code. I'm at 170K lines of code in this codebase without reading any code.

Reliable Systems from Unreliable Parts

You've probably seen some hallucinations, or just terrible results in your vibe-coding attempts. That doesn't mean all is lost though. Here's an analogy:

TCP is a reliable protocol built on top of an unreliable one. IP packets get lost, arrive out of order, get corrupted. But TCP uses redundancy, acknowledgments, and retries to create reliability from unreliability.

The same principle applies here. Individual AI agents are unreliable. They hallucinate. They make mistakes. They miss things. But multiple agents, possibly from different vendors and models, tasked with bringing different perspectives -- that's how you get to a more trustworthy solution.

There are consensus skills for Claude that do exactly this. Multiple agents review the same thing and have to reach agreement. If one agent hallucinates, the others catch it. Unreliable pieces used together to create reliable systems.

This is why I'm not worried about AI reliability for code review. We don't need each individual agent to be perfect. We just need the overall system to be reliable. And we know how to build reliable systems from unreliable parts. We've been doing it since TCP. I wrote more about this in Probabilistic to Deterministic.

Making It Work

A few practical things:

Reviews should happen precommit. Anything the review finds should be added to Claude -- CLAUDE.md files, custom slash commands, system prompts. If a review catches something once, your agent should ideally never make that mistake again. Put a feedback loop on your feedback loop.

Organize code for AI. Smaller files that fit in context windows. Clear naming that makes intent obvious without needing deep context. We used to organize code for humans. Now we organize it for AI too.

Lighten up your dev environment.

Well this is pretty wild: Now my IDE is code-free.

I use zellij to tile my windows, put claude code cli in the middle, with a few running terminals on the right. For git I love the auto-updating views that IDEs give you, so I vibe-coded a quick terminal app that just gives you that view. It took about three prompts to create and two of them were to suggest a good name. But there's nothing for viewing / editing code.

Conclusion

I don't at all think we're living in a post-code age (yet). But I think code reading and writing by humans might be over. To be clear: I definitely still need to know how to do software engineering to write anything complex beyond a one-shot, but of course coding is only part of software engineering, and it turns out it's a small part.

I have no idea what this means or what's next.

The Reliable Org

2024-10-14T08:00:00Z

My software engineering background colors my management style in many ways.

Here’s an example that’s a cornerstone for my definition of good management:

Most people in software know a “200 OK” is a web server response that says that the request was successful, or a 400 Bad Request error that something went wrong. Less common is a “202 Accepted” response, but it’s worth explaining: A 202 Accepted response means “I got your request, thanks. I’ll work on that.”

An endpoint that uses 202 is one that I would call “unreliable”, and that’s generally a system property to avoid with all else equal, just like it sounds. An endpoint like that gives you no idea if your request will ultimately work -- just that the system accepted your request. With an endpoint that returns 200 OK or an error like 404 Not Found though, you get to find out and find out right away.

When talking about whether an endpoint is “reliable” or any system or component is “reliable”, people usually think it means that it’s really good or perfect at doing its job.

“Reliability” from a distributed computing perspective means that there’s a way for the system to communicate failure, not that it works really well. An endpoint that gives 202 responses would be considered unreliable even if it had a 0% failure rate over a decade. This is a critical difference because when a component can communicate problems that it encounters, you can do things like fix the component, or retry the component, or switch to a different similar component. That is to say you can manage the component.

Often less experienced engineers think that this definition of reliability doesn’t really matter as long as the component is just perfect at its job all the time, but a more robust system can be built by understanding that perfection is impossible, and instead we should build a system of components that communicate when they fail so we can properly manage the situation when inevitable failures do occur.

So what does this have to do with people management? Well people and teams and divisions and business units are all components of the larger system of the business. Just as reliability is critical in distributed systems, it plays an equally important role in managing teams and organizations. What does reliability by this new definition mean for systems of people? It means that people communicate in a way that meets our new version of reliability: they can and do tell you when things are not going well.

Reliability can usually simply be built into a software system, but for humans that bring a more emotional aspect to the system, you can’t get reliability that easily. You need to consider multiple factors to make a system of people reliable. I’d like to chat now about the hardest one to get right and the one that most organizations lack: psychological safety.

Psychological safety is when people are free to voice opinions, disagree, take risks (and fail), and make mistakes without fear of punishment or other negative consequences. An organization that can freely say when things have gone wrong is one that can be considered reliable in the same way that technical components that communicate failure can be considered reliable.

Most companies don’t care about psychological safety. Here’s what they do instead:

Look for someone to blame when things go wrong
Push pressure and goals down a hierarchy away from themselves shirking responsibility
Cover-up mistakes and hide failures.
Pursue conservative goals instead of game-changing breakthroughs.

I’ll give you a real world example: A friend of mine is a head nurse at a prestigious hospital. She discovered that another nurse had accidentally given an incorrect and dangerous medication to a patient and didn’t know if she’d be fired or reprimanded. Of course my friend corrected the situation immediately, but things could’ve gone a different way pretty easily. Additionally upon further investigation, the dangerous medication was stored immediately next to the correct medication and had all the same colored labels. Management can’t reduce the instances of cases like this if they don’t know about them, or the details of their causes, and if they can’t think beyond blaming the person that gave the wrong medication.

Blame makes people hide things. People hiding things makes them unreliable.

Now you can probably think of a million other examples of this in real life. Have you ever heard of an aviation disaster where they conclude “human error”? That’s another ridiculous example of complete lack of psychological safety. If they stop the investigation at “human error”, and still rely on humans in the airport, air control, and on planes, then I don’t know about you, but I’m personally pretty terrified for exactly the same thing to happen again. If they instead assume that the human did the best they could in the situation and that we should find another way to improve our chances next time, then they’re actually on a path of learning and improvement.

You have to choose between the ability to blame people and the ability to learn and improve. You can’t have both.

For this reason an organization that has psychologically safety is an organization that:

Learns
Improves
Is reliable
Is manageable

Once you see it, it’s hard to consider any other type of management as management at all.

Breaking It Down, Part IV

2022-05-23T12:07:00Z

(special thanks to Deepa Joshi for her input!)

This is the fourth and final part of my series on work breakdown (WBD) for large software efforts. In the last post I chatted in detail about what constitutes a good piece of work in the breakdown, but in this post I'd like to get more holistic and talk about what the entire WBD will look like when it's done well, and how the pieces of it should inter-relate. This is the critical big-picture view.

Properties of a Good WBD:

A good WBD will ideally have 3 primary properties defining the relationship between the pieces: mutually exclusive, collectively exhaustive and prioritized.

Mutually Exclusive

The pieces should ideally have no overlap of effort between them. You haven’t really successfully broken something down if the pieces haven’t been completely separated. Some pieces will definitely require other pieces to be completed first, but that can be done with proper sequencing and without actual overlap.

Collectively Exhaustive

The pieces should ideally comprise the entire effort and not just part of it. It’s much more difficult to manage an incomplete WBD where we’re regularly adding more work. Many runaway projects that I’ve seen suffer from a lack of holistic view of the entire breadth of the project. Sometimes there are a bewildering number of unknowns, but those should be handled as early as possible with questions, experiments, prototypes, research, etc, so that the WBD converges on “collectively exhaustive” as quickly as possible.

Prioritized

Prioritization is another huge part of the breakdown process. If we’re continuously delivering value, we want to be delivering the highest value first and then decreasing value over time. There are different kinds of value to consider too: derisking, dependencies, learning & business-value. You also want to consider the cost of them (usually just measured in developer time, but not always). I’m not going to get into prioritization frameworks, or tell you how to decide between the things that deliver business-value, but it is useful to talk about how to prioritize business value among the other kinds of value.

User / Business learning: Probably the biggest waste in software development is building the wrong thing, so we want to prioritize this type of learning pretty highly. Especially learning that:

stops us from building the wrong thing
Informs us about what we should be building

Approach-Derisking: Assuming we’ve got high confidence in what we’re building, we next want to derisk our approach. First we want to derisk the actual plan by figuring out whatever we need to, to ensure the work breakdown is comprehensive. Secondly we want to do any technical research, prototypes, or experimentation that we need to do to clear up anything that’s technically complex and not straightforward.

User / Business value: You want to deliver things in order of business value so you capitalize on your efforts in the most optimal way possible. Even if you’re not delivering to production until completion of the entire effort, you still want to derisk the highest value things first by getting them as complete as possible first.

Dependencies: You’re basically forced to prioritize fulfilling dependencies before their dependents, but it’s helpful to make this explicitly part of your prioritization. Otherwise there’s potential for people to not see dependencies and start work on something with unfulfilled dependencies. They’re sure to eventually get blocked and forced to work on the dependency. Dependencies themselves have risks as well, so it’s better to work on them first and not have them block other efforts.

It’s also worth noting that the clock runs out on plans before they’re complete all the time. Companies change priorities and cancel plans in progress. Prioritizing like this ensures that the work that we don’t do is the lowest value work, and ensures that even if we’re interrupted, we’ve still captured as much value as possible in the time we had. It’s also worth looking at the lower priority work early-on to see if that work can just be considered for a subsequent effort instead, or even eliminated. Proactively removing work that doesn’t need to happen as part of this effort can have a massive positive impact on the size of the effort and the speed at which it’s delivered.

Where Possible, Begin with Deployment

Being able to build and deploy the work is table-stakes. You can’t play the game of software delivery if you can’t build and deploy the work. I highly recommend that the first thing that you do is to ensure deployment is possible and safe.

For new software where no one is using it yet, it’s good to set up your build and deployment so that you ensure early and often that what works on your machine works equally well in your target production environment. It takes no additional time because you have to do this eventually anyway, and you don’t want to be debugging environmental differences after you thought everything was done.

For existing software, ensuring safe deployment is equally critical. If we’re focused on continuous delivery and we’re so results-oriented that our progress is visible in production, we’re almost certainly using feature flags so it’s worth chatting a bit about them as part of the WBD for those that are working this way.

My primary piece of advice is that you always deliver the feature flag FIRST as part of the greater effort, because that’s what enables the rest of the delivery to be both safe and continuous.

Secondarily, it should reveal as small a piece of functionality as possible to be still visible. The functionality revealed could just be a placeholder for future work (like a “hello world” message), but you do want to show that the feature flag is working end to end (be Visible!). This proves that you will be continuously deployable as you go, and allows you to turn on the feature flag internally so everyone can test the real production progress whenever they want.

I think this feature-flags-first policy is worth noting here separately because in general people naturally tend to not add the feature flag until the perceived end of an effort when they start thinking about delivering everything in totality to real users. Adding the feature flag last means that you’re integrating your work at the very end. That’s a completely unnecessary addition of risk. Your internal teammates have a harder time seeing things along the way and giving feedback, and you unnecessarily delay finding out about any trickiness of adding the flag that might seriously impact your timeline. Some feature flags are really non-trivial to add.

Continuity

Finally, I’d like to discuss how to do all this continuously. A good WBD is the essential core of a good plan in general. The thing to know about plans though is that they quickly drift away from being the most optimal plan as we work through them. We learn new things. We find out that we had some things wrong. Circumstances change. Plans fall apart.

For these reasons my strong recommendation is that, as you go, you continue assessing your work breakdown along all the criteria written here, and improve it continuously. You don’t want to foolishly pursue a plan that is falling apart. You want to react to new information and keep adjusting the plan.

The most common questions you’ll want to ask yourself are:

Do we still have a comprehensive view of the work? Or is there more essential work that needs to be added?
Are we still prioritizing properly? Did we find new risks that we should derisk as soon as possible?
Is there anything we can take out of the work breakdown and do another time instead?

In short, make your planning and replanning activities more continuous. Ask yourself continuously: Is this still the best plan? If it’s not, fix it!

In Conclusion

I know that this 4-part series about breaking large software efforts down can seem like a bunch of bureaucracy compared to immediately getting to work on writing the code, but it really is pretty critical if your software effort is large. Sure I've seen teams function well without this kind of rigor, but usually it's when they have work that is already naturally broken down into thin vertical slices (eg maintenance teams or growth teams). For other types of teams that actually are delivering larger efforts, not putting the time in to get a good WBD can mean that you're working on the wrong things more often, or delivering value to users more slowly. In the worst cases, the project can go completely off-track and be a total failure. Not only will a good WBD help prevent these types of things, but sometimes you can even apply a good WBD to a failing project and bring it back on track.

Think of it instead as a design exercise. You want to design your work so it's the kind of work that makes you look and feel successful as you accomplish it. It also happens to be the kind of work that will actually be successful.

Breaking It Down, Part III

2022-05-19T15:46:00Z

(special thanks to Deepa Joshi for her input!)

I introduced the concept of thin vertical slices for work breakdowns (WBD) in my last post but the idea is by no means new, and I didn't invent it; thin vertical slices have been the state-of-the-art for work breakdown (WBD) for decades. People often don't spend time studying the state-of-the-art though, and they're more comfortable breaking things down along more natural boundaries, so a lot of teams don't actually break things up along vertical lines. I think ignoring this best practice either hurts or dooms a lot of larger software efforts though, so I'd like to just go over the subject again here, even though this a pretty well-trodden concept.

Thin Vertical Slices

Ultimately each piece in a WBD needs to have certain properties in order to really meet our goals. Those properties are Vertical, and Thin. Let’s dig into them:

Vertical

Breaking things down along vertical lines where “Working software is the primary measure of progress” gets us both Visibility (true visibility!) and actual Value Delivery. Also if we’re succeeding at visibility and value delivery regularly, we get a feeling of momentum for free, and so Motivation is easy to maintain as well.

This can be hard, because it means a particular piece of work may need different developers to make sure you tackle all the necessary horizontals or components in it. That’s fine! Multiple people can coordinate around a single piece of work. That’s the price we’re willing to pay to ensure Visibility, Value Delivery, and Motivation.

Thin

Now if you manage to break everything down vertically, you can actually slice the work into ridiculously small pieces and enjoy completing things continuously all day long. That’s really energizing and people watching the progress are generally getting a very real (and impressive) view of what’s happening and how fast things are moving.

More importantly, breaking things down into smaller pieces that everyone agrees on is the best way to get everyone’s more-precise agreement on what we’re trying to do. The more pieces there are, the less chance that there’s some large hidden detail that we disagree on. Of course, once you get started the plan tends to fall apart a little bit (and sometimes a lot), but the discussion and documentation of the plan tends to still be valuable because it creates alignment around what you’re trying to do in increasing amounts of detail. That alignment will be invaluable for recognizing when the plan isn’t on track and how to fix it.

There is a balance to maintain as well. It is possible to over-define a plan; if something is trivial and multiple possible solutions are roughly equivalent, planning every detail out can yield diminishing returns. Planning itself takes time. If your planning activities are not significantly reducing risk anymore, you can stop planning and get to work. If you’ve ever been on a team that debates for more than 30 seconds about how to prioritize a typo fix, you know the pain of over-planning.

Slicing your work into small pieces also makes it easier to estimate. Small estimates simply have less error in them than larger estimates. Also if your pieces are small enough, you can just count how many you tackle in a week to get a rough estimate of when you’ll be done.

Sidenote: Dealing with Uncertainty

It’s worth noting that some WBDs will be hard to do comprehensively because of unknowns. If you break things down into small pieces, it’s likely that they drive out many pressing questions / unknowns. It’s totally okay to create work items for experiments, prototypes, research, etc that may affect the direction of the rest of the work. They may or may not have the same definition of done as usual production quality software because the goal is learning or derisking and not user-value.

For example, if you’re not sure how much work something is, and want to choose a different solution if it’s really expensive, you might want to create a separate timeboxed piece of work solely for that investigation, eg “Hey I need 2 days to evaluate some libraries to see how hard that is”. When the timebox expires we hopefully have an answer that we can work with, or we can decide if we should do another timebox. A few subsequent failures of a timebox to yield our answers is itself an interesting data point -- maybe this thing is just more complex and expensive than we’re willing to take on. Or maybe everyone agrees on more timeboxes still.

In general, activities like these that make the plan clearer and help us derisk our efforts are worth doing first and in order of risk so the plan gets clear as quickly as possible and the work gets more and more straightforward.

Next Up

Alright! Now that I've talked ad nauseum about the pieces of the WBD, it's time to talk about the properties of a good WBD as a whole. The next and final part will get into that critical and rarely discussed aspect of WBDs.

Breaking It Down, Part II

2022-05-09T08:15:06Z

(special thanks to Deepa Joshi for her input!)

I previously explained why you should care about a good work breakdown (WBD), and gave some examples of common ways to do work breakdown that we should try to avoid. None of that is much help unless you know what we’re really trying to achieve. I’ll explain that here.

The whole point of the WBD is manageability. We break work down because a single big 3-month effort is just too difficult to manage without decomposing it and thinking of it in parts. It’s just too much to fit inside any one person’s head at once in its entirety, so it’s easy to fail at managing the parts we overlooked or forgot.

We can organize the WBD in additional ways that improve manageability as well though. We want to make sure things stay as closely to our idea of “on-track” time-wise as possible. When something is off-track, we want to be as obvious about that as possible as well so that we can do something about it as early as possible. Manageability is about easily determining if our approach is working or not, and adjusting.

In short, we want to ensure real progress and immediately see when we’re having difficulty progressing. People need to know about progress in the software development process so that they can determine…

…how to help.
…what to do next.
…how to set expectations and tell others.
…if we’re meeting timeline expectations or not.
…if an effort is stuck and might need external help or a different direction.

In general, there are 3 primary factors that help us ensure progress: Visibility, Value Delivery, and Motivation. Let’s dig into these factors.

3 Factors For Ensuring Progress

Visibility: Is what we’re trying to do obvious, concrete, well-shared, and well-understood? Can people (including us) see our progress? Does what they see along the way accurately reflect real value being delivered? As pieces are delivered, do we all get more confidence that we’re on the right track? Do we have all the work enumerated so that we’re not adding more work at the last minute? Are we managing the expectations of stakeholders? If we get interrupted partway through, did we do the best we could have done with our time? If we’re doing things that we don’t need to do, will other people be able to see that so that they can tell us?

Value Delivery: Are we derisking early? Are we learning early? Are we delivering the highest value things first? Are we delivering the lower value things later, or successfully saying no to them? Are we getting things to users as frequently and as soon as possible? Are we providing maximum value at all times? Are we learning as much as possible along the way so we can continuously improve the remaining plan?

Motivation: Are we tackling scary unknown parts quickly so they don’t hang over our heads? Do we feel like we’re making progress and getting momentum? Are we delivering regularly and feeling “in the flow”? Are we reducing anxiety by making sure that there’s no point where we have to communicate a big negative surprise to stakeholders? Are we tackling risky integrations early and frequently to avoid a bunch of rework?

If we’re going to maximize these 3 factors, we need to break the work down in a particular way.

Working Software is the Primary Measure of Progress

Sure, “Working software is the primary measure of progress” is the 7th principle of the Agile Manifesto , but here it’s worth learning and following because it gets us our goals of Visibility, Value Delivery, and Motivation so much better than anything else I’ve seen.

Why is working software the measure for “progress”?

Anyone who’s written a lot of software knows that a bunch of things can go wrong at different stages of the software development process. Let’s talk through three of those stages:

Getting your change into main/master/trunk:

It’s difficult to merge because of changes by others in the same files (merge debt).
Fellow engineers have significant concerns about your solution/code.
The people you need to review it aren’t in today.
You break the build somehow.
Your work doesn’t have merge conflicts, but also doesn’t fulfill a working contract with the larger system (eg “We’re done the backend. We just have to do the frontend.”).

Showing PMs/designers/testers/stakeholders:

We didn’t build what we thought we agreed on.
We built it exactly how they asked for it, and after having seen it / used it, they realize it’s not as great of an idea as they originally thought.
They find defects.
The people that need to sign off on it are on vacation.

Going to a Production environment:

The prod db has some data we didn’t expect.
The feature doesn’t work at production scale.
Users don’t like what we built.
There’s a memory leak when we run it for a few hours
We didn’t get the production instrumentation (logging / metrics / alerts) quite right

These are all the reasons that no one should really be all that excited when developers say “it’s done but I just have to…”

Merge it into main/master/trunk
Get it code reviewed
Test it / get it tested
Show it to the designer / pm / tester for feedback
Deploy it / release it

There’s just so much that can still go wrong that saying you’re done is incredibly misleading to yourself and to others. Many of these things that can go wrong could even require you to have to start again entirely.

This is why we say “working software is the primary measure of progress”. If the thing is working in production, ideally in ways that are verifiable by non developers, and even more ideally in ways that end-users can verify, then we can prove to ourselves and others that we’ve passed all these stages of risk and something is actually done. Delivering as close as possible to the real production environment also leaves very little opportunity for miscommunication about the progress; anyone should be able to go and see it for themselves.

The brutal truth about software development is that until your code is actually working and in real actual use, you've only really managed to cost your company time and money.

In general, you should embrace the most risk-free and comprehensive definition of done that you possibly can so that when you say “done”, it really lets others know that:

You don’t need to work on it more.
It’s ready for people to use.
We’re achieving the results we wanted.

This is what everyone really needs to know. All the previous stages still contain too much risk for you to be saying “done”.

As a result, work should be broken down in ways where each piece:

…can be validated/enjoyed by all interested humans including ones that are as close to real users as possible.
…can be delivered as close to a real production environment as possible.

Outcomes > Outputs > Efforts

We want to be as outcome-oriented as possible, so that the progress we’re showing is as real as possible. My shorthand for this is “Outcomes > Outputs > Efforts”, which means that while we appreciate efforts, we appreciate outputs more. And while we appreciate outputs, we appreciate outcomes even more.

We appreciate efforts like “coding” and “testing”, but not as much as we appreciate an actual output of the efforts like a shipped feature that we consider production worthy. Similarly, we appreciate outcomes like “the users love the new features”, or “revenue is way up as a result of the features we built” or “we learned a lot from that latest experiment we shipped” even more than we appreciate the output of shipping the features. So when we break work down, as much as possible, we want to avoid breaking pieces off that describe efforts, and ideally break pieces off that describe outcomes as much as possible. Most times your pieces should at least describe an output.

This is the reason that, for example, we generally don’t want “testing feature whatever” broken out as a separate work item from the rest of the work. If the act of testing is extracted from the rest of the work, then the remaining work can never really meet a very meaningful or useful definition of done. “Testing” is just an effort -- it’s very far from an actual business outcome. It also forces the remaining work items to not include testing as well, relegating them to effort status as well. Additionally, testing can find that the other work is broken in ways that invalidate it. Ideally when you break work up, the pieces are maximally decoupled and can’t invalidate each other like that.

Focusing less on effort and more on results is also the reason we don’t want work broken up along what we call horizontal lines. For example, if you were to split the frontend work and backend work into separate pieces (even if the frontend work is in a different codebase, or even still if it’s worked on by different people entirely) that would mean that the separate efforts are never independently verifiable or usable by people as close to the user as possible in an environment as close to production as possible.

Instead we want to slice the work in vertical pieces. Imagine how you’d share a hamburger with someone else: you’d cut the burger vertically so they get bun, lettuce, tomato, burger, and bun again, all at once in one slice. You would never slice horizontally and give them the top half. Similarly, you want each piece of work to deliver a portion of the user’s experience, and not some horizontal slice like “the frontend” or “the backend”.

Also notice that if our work is always broken-down along vertical lines, it’s easiest if it’s now broken-down into extremely small slices. The thinner your vertical slices are, the easier/faster it is to get any one of them through all the risky merge/validate/release stages we talked about earlier. Even more importantly, if something fails to get through one of those stages in a way that requires a dramatic change of plan, you learn about it as early as possible, instead of after building a bunch of software that doesn’t actually achieve the outcomes we want.

In my next post I'll dive a bit deeper into thin vertical slices, because they're so critical and so hard to get right.

Breaking It Down, Part I

2022-04-30T11:51:06Z

(special thanks to Deepa Joshi for her input!)

A critical skill for senior engineers is knowing how to break down a bunch of work into smaller, more manageable pieces. You need to make sure you’re spending your time and effort on the right things, and that the rest of the organization is in agreement. This isn’t always the most exciting part of software development, but I’ve seen a proper work breakdown (WBD) shave months off a project, even when done after the project is already months late. I’ve never seen it be a waste of time, no matter how anxious people are “to get to work” (Those are semi-sarcastic quotes, because planning is part of the work of software development).

Of course, the WBD is no big deal if your work is regularly simple and small and involves few people, but for anything more complex, not doing a proper WBD usually leads to lots of problems like:

The larger organization feeling like it’s in the dark not knowing if the effort is on-track
The developers themselves not realizing that they’re working on the wrong things (and the entire effort actually is off-track!).
The project gets more off-track and there’s an increasingly difficult conversation to have with the larger organization about how the effort is failing.

The WBD is critical to starting the effort on-track, keeping it on track, and even putting it back on track when it starts to seem out of control.

Bureaucracy Is A Lot More Fun Than Failing

If you’re anything like me you just want to get back to work and don’t really want to be concerned with this kind of bureaucratic bean-counting. It’s certainly not much fun, and it can feel like a waste of time to spend so much time talking about the work instead of getting started on it. Try to keep in mind though: wasting your time building the wrong things, getting results way slower than you could be, getting mounting pressure and criticism from an organization -- all of those things -- are even less fun than doing a proper WBD. A proper WBD is pure bliss compared to a large project failing. You’ll never be a useful contributor to the management of a larger software effort unless you're willing to learn how to do it.

Your Intuition Won’t Cut It

A lot of developers have a bunch of experience doing WBDs along natural boundaries into smaller pieces and maybe have seen some success, especially on smaller projects. If they haven’t studied what other people in the industry are doing though, they’re almost certainly not doing very effective WBDs. Good WBDs don’t look at all like you would naturally or intuitively expect. There are decades of prior thought on this and you’d ignore it at your folly.

Here are some common methods of breaking work down that don’t work very well:

Splitting Along Architectural Lines

It’s important to not confuse a work breakdown with an architectural design. You might have various architectural components involved in a larger effort you’re undertaking. They could be standard architectural components like model/view/controller or just frontend/backend or multiple services in a more service-oriented or microservice architecture. It’s absolutely worth a good chat about the architecture, and it’s worth doing an architecture breakdown with diagrams and written plans. I would keep this very light in the beginning and define it as you go, but it’s definitely worth doing.

Don’t confuse this with a WBD though!

A WBD is intended to be something that makes sense to non-engineers with the ability to make progress visible. Most architectural components make no sense to non-engineers, and deliver nothing on their own to users. You actually can’t even honestly say to yourself that an architectural component is done with certainty unless it’s integrated and in use by its collaborating components. Instead we want to drive uncertainty like that out as early as possible.

The Smallest Pieces Possible

In interviews when I ask Senior Engineering candidates how to break work down, the most common answer I get is “as small as possible”. This is a nice answer insofar as it’s obvious that they do value breaking down larger pieces of work in general (rather than a single 3 month task!), but probably the smallest possible breakdown is character by character of the source code. I’m certain that no one actually means that.

If you’re doing the smallest pieces possible, very few of those pieces will deliver anything to users at all, or result in any learning for the company. You can work for weeks without contributing to progress visibility or value delivery. Of course we want to divide-and-conquer the larger effort, but we’ve got to be more thoughtful than just saying “the smallest pieces possible”.

Instead, we want the smallest pieces possible that still make our progress obvious (ideally even delivering actual business value). If you break things down further than that, you run the risk of showing false progress, and introducing many new kinds of risks.

Work that can be assigned to individuals

The next most common way of breaking work down seems to be for the purpose of assignment to different people on the team. This is where you get tickets for the frontend devs and tickets for the backend devs, or one ticket for the devs and one ticket for the tester.

This can be a nice clear way of showing who is working on what, but it actually hurts both visibility and value delivery. You get people saying “it’s done but it just needs to be tested”, or “the backend is done but we still need to build the frontend”. These are weak definitions of “done” where we can’t deliver anything to users, and we’ve learned nothing in production. Saying “done” in these cases is a bit of a smoke screen to stakeholders; the testers could reveal bugs that require more work, and the frontend devs could uncover missing functionality on the backend. Without them being in a production environment with real users you still won’t know about the defects that the testers missed, or whether your backend can handle the load. Telling the stakeholders that something is done in any of these cases won’t really be 100% true and so at best won’t be as transparent/visible as you could be. Sometimes I’ve seen people showing a frontend on its own and that can be downright deceptive to non-technical folks that don’t realize how much work there is involved in making the backend (which is sometimes the majority of the work).

Note: Unfortunately most ticketing systems only let you assign a piece of work to a single person, but you shouldn’t let the tool get in the way of you doing the right thing and having multiple people (where useful) working on the same ticket when it’s the most visible way to work and gets you actual value delivery. You can easily chat informally (off the ticketing system) as a group about who does what if you want to take different portions of the ticket individually. If you need to put a single name and face on a ticket, choose a leader for it and assign it to them.

The Cliffhanger

Ultimately a good WBD should put the focus on real accomplishments as much as possible instead of efforts, and put the focus on what the team is doing, rather than what individuals are doing. How do you do that? I’ll tackle that in my next post.

Hey that seems longer than 2 weeks

2022-04-28T11:50:06Z

I’ve stumbled upon a style of estimation that I’ve seen be pretty successful. I’ll call it “2W+ Estimation”. I've mentioned it before, but without much detail.

There are two possible estimates given for any work that the team is about to take on: “more than 2 weeks” and “less than 2 weeks”.
An estimate is not provided at all if the work seems like less than 2 weeks. It is assumed to be less than 2 weeks. We only mention an estimate if we want to note that something seems longer than 2 weeks.
When something is bigger than 2 weeks of effort, we try to think of ways to change the scope to something deliverable in less than two weeks while accomplishing the same goals. Sometimes we talk about whether the high cost means it should be prioritized lower. Sometimes we agree it’s worth doing now anyway. Sometimes we break it down into valuable deliverable pieces, where each is smaller than 2 weeks.

It’s worth noting that there are 2 critical steps that we undertake afterward for any effort large or small: (1) break it down into thin vertical slices (2) check every day to make sure things are still tracking how we’d expect. Bigger-sized projects tend to be riskier and go off the rails more often. If one seems off the rails we might have to restart this process and reassess the remaining effort.

What’s so great about this estimation style?

It’s almost as fast as not estimating at all, but forces good conversations in higher risk scenarios.

It uses time not complexity (like story points) because (1) everybody already knows and understands time and (2) complexity is not any easier to estimate.

No overly precise estimates are ever given, so the estimates can’t be converted to precise calendar expectations. We don’t like the effects on our codebase or people of them being rushed with arbitrary deadlines.

Why 2 weeks?

I imagine a different time frame could work better in different scenarios. I even imagine a smaller time frame could work better for us.

How did we develop this?

We started by not estimating, and then smart people who found that too risky would pipe up pretty regularly and say “hey you know that’s pretty big right?”. “Pretty big” seemed to be generally agreed to be “greater than 2 weeks”.

Who might this not work well for?

Well you probably already know if you’re at the kind of company that would even be willing to try this. A lot of companies think spending more time on estimation gets better estimates, and that more precise estimates are worthwhile. We don’t, but different companies value different things and sometimes those values have good reasons.

The team needs to be willing and able to speak up when something is large because there’s no other mechanism than that kind of proactive candor.

This may not work for a team that does sprints where you're trying to pack as much work into a given timeframe as reasonable, especially if those sprints less than or equal to 2 weeks. We find sprints to be really weird, so we don't do them. T-Shirt sizes might be better for those kinds of teams, or something even more precise.

You still need to break stuff down well and reassess progress daily. But everyone should be doing that regardless of estimation technique.

Sometimes TDD

2022-01-19T11:50:06Z

It can sometimes feel like TDD is pretty dead in 2022. I’m not sure what happened, because I think it’s a genius way to work, but most programmers I talk to seem to think that it’s a waste of time. We don’t really have to be dogmatic about it though. It’s not equally valuable at all times, and there are even rare times when a total kool-aid drinker like me doesn’t do it. This post is meant to chat about the extremely high-value times.

For defects

I work with a lot of people that almost never do TDD but a few seem to pull it out a lot when they’re fixing defects. It makes a lot of sense. When you’ve got a defect, you’ve first got to make sure you know how to reproduce it. If you write that code to reproduce it first, you’re 90% of the way to doing TDD. Add an assertion to validate that your app code does the right thing, and then fix your app code to pass the test. This really makes you feel like you’re making your code bulletproof and you won’t have to hear about this particular defect again.

For a complicated problem

I’ve had a few examples of complicated problems that would have been almost impossible (for me) to solve without TDD. I once wrote a kafka broker for node.js that ended up being the only Kafka broker in node.js at the time, used by a few big-named companies. The protocol was all binary and I worked from the protocol documentation to write tests first, and then the code to pass the test. I had my test environment hooked up to a real kafka server so that when a test passed, I knew it really worked. I slowly worked my way through the protocol documentation this way until it was done and working. I did a ton of refactoring along the way and a ton of refactoring afterward as well, and the tests gave me confidence in my changes and a very tight feedback loop. Whenever I can tell that something is going to get really complicated, I switch gears to slow, methodical TDD.

For something that requires more “architecture” than usual

There are a lot of people that I’m not going to sell with this point, but it’s been the biggest thing for me with TDD. If you’re working on a codebase that’s going to grow larger and larger over time and be worked on by many people, you need to think about the internal organization of that codebase (“architecture”) if you want to keep it from quickly becoming a big ball of spaghetti. Not only do you need to think about architecture, but you also need to think about the testability of that architecture regardless of whether you write the tests first or after. Driving out architectural pieces using TDD ensures that things are factored and working together in a way that is testable. In this way the last D in TDD is more like “Design” than “Development” because the tests have you architecting the codebase differently. Some will argue that it’s not the beautiful design that they would like, but I’ve often found that the factorings are quite elegant and that testability is more important than beauty anyway. In my experience, when you’ve architected like this, you can bring on the hordes of test-later programmers pretty successfully and get great tests and great coverage. In my experience, a codebase is testable much more easily if it’s TDD-able.

For something that’s untestable

If you’ve got something that’s untestable or hard to test and you’re struggling to get tests on it, it can feel like it would be even harder if you had to write the tests first. In general I find the opposite is paradoxically true, and this is another case where the last D in TDD is more about “Design”. Forget about that working code that you wrote and just try to write a small test like how you wish a test of it could look in the simplest human-readable way. Now see if you can write code that can pass it, ignoring your previous app code. Concentrating on the test first drives out a testable design and I believe you’ll almost always like the factoring as well. Sometimes the test is the hardest part, so starting with it can be the best way to make sure you nail it.

Anyway, these are just my opinions after a decade of TDD and working with many others that seem to only use it occasionally. I still think TDD does a better job of getting tests and well-factored code in 99% of scenarios, but these 4 are the truly magical scenarios for me. My advice is to give TDD a chance in these cases if you’ve given up on TDD entirely or if you never saw the point.

Block-Testing

2020-11-02T02:45:14Z

How you do QA is probably terrible.

Of course I don’t know you, but most software development shops that I’ve worked on or chatted with “do QA” in a way that is pretty terrible.

Almost everyone does it, and almost everyone considers it a “best practice”, and yet somehow it never even got a name. I’m going to call it “block-testing” because it’s the kind of testing that unfortunately makes continuous deployment and continuous delivery fundamentally impossible.

So what is it already?!?!

It’s the practice of having humans try to regression-test an entire product or some high-value subset of features across the entire product usually with the goal of finding reasons to stop it from being released. The most extreme (worst) case of this being when a specific role exists solely for this purpose (almost always given the misnomer “QA”).

Just to be super-duper clear: I’m aware that almost everyone is doing block-testing. That doesn’t mean it’s a good fit for every organization though. If you actually want to achieve continuous deployment or continuous delivery, this is one of the big-batch operations that you’ll need to forego.

What’s so bad about it?

A single manual run-through of a broad test script is slow (compared to existing automation)
It’s expensive (compared to good automation, over the long-term)
It frequently misses all but the most egregious regressions (more often than alternatives)
It’s antagonistic, pitting different roles at the company against each other.
It confuses where the responsibility for quality lies
There are many other possible solutions to the same problem, many of which are faster and more comprehensive.

Alright a couple of those things are obvious to most people I think so let’s just dig into a couple of the less obvious ones:

It’s antagonistic

Hiring people whose sole purpose is to find reasons to stop a release is counter to the goals of a company that is trying to release software quickly. Of course block-testers are actually tasked with the important goal of making sure each release is high-quality (and that’s a critical goal!), but the end result is that as block-testers, they naturally tend to optimize their role to “trying to find release-blockers”. They’ll log other defects too of course, but those don’t have the same importance to them (or anyone else) as issues that block releases.

Also when you’ve got one group of people responsible for finding fault in another group’s work, you’re going to have conflict between the groups. Engineers are going to start saying that the bug isn’t theirs or the bug isn’t “that bad”. Testers are going to say the opposite. Each group is going to blame the other when management wonders why releases are delayed or infrequent. You’ll get the same finger-pointing when a defect gets through to production as well. The ensuing arguments are a massive waste of time and each “side” becomes less empathetic to the other’s plight over time.

It Confuses where the Responsibility for Quality Lies

I’ve heard the words “You’re the tester! Test it!” a few times in my career. The sentiment is that development and testing are distinct phases rather than something that happens iteratively during development, and “the developers do the developing and the testers do the testing”. “This frees developers up to just focus on just development”.

Nothing really could be further from the truth and I have some serious doubts about the experience of any developer that thinks that they shouldn’t be testing their own work along the way as if they can hammer out 15 or more LoC and won’t have to bother to try and see if it works. The fact of the matter is that quality is entirely the responsibility of the developer and any successful developer is testing constantly as they go. Conversely, the tester’s only role is to assess the quality. The tester has no responsibility whatsoever to improve it.

At the same time, the very presence of the tester is a very confusing signal. It signals that the developers aren’t wholly responsible for quality (and maybe even that they’re not trusted with it!). It diminishes their skin in the game, and disincentives quality. Of course a tester-only role has no ability to actually fix defects, and you can’t sanely hold someone responsible for something they have no control over.

I would wager that this division of responsibility is the single greatest cost of block-testing. The testers and the developers start to engage in a kind of game of ping-pong where the developer tries to see what they can get past the tester, and the tester tries to find ways to block the release for any reason at all. I’ve seen a feature go back and forth from developer to tester more than a half-dozen times before the tester agrees that it’s okay to release. That back-and-forth often plays out over days through an issue tracker for trivially-fixed-but-important issues in high value features. If you’re looking closely for this kind of activity you can see it all over the place, and it can look about as effective as a football team that uses telegrams to communicate mid-play. These hand-offs are all classic waste from a Lean perspective.

There are many other possible solutions to ensuring quality, many of which are faster and more comprehensive.

Once you realize the harm, the cost, and the inadequacy of block-testing, you naturally have to ask, “but is there anything better?”. You can’t really fault an organization for block-testing if there are no alternatives.

Let’s chat about some alternatives:

Move Fast and Avoid False Dichotomies

It’s really unfortunate that the industry has resolved itself to the belief that you have to either go slow or you’ll break things, and that quality and speed are an inevitable and universal trade-off. This is a really limiting belief.

In order to get creative, let’s forget the “best practices” for a minute and get back to first-principles: Here’s what we’re trying to do:

We want to minimize users’ exposure to defects and outages.
We want to minimize the time spent between a developer completing a change on their workstation and it getting to users to keep delivery radically fast.
We want to catch more issues than a human can.

Preventative Measures

Preventative measures are anything that you might do to prevent a defect from affecting a user. They’re a pretty critical part of achieving goal A.

However, goal B means we really have to be careful what steps we put between the developer and the production environment. Ideally they’re critically important and they’re fast. Each of these steps has to pay for themselves. Because of this, you have to be really careful about what preventative measures you choose. Some really fast preventative measures: static-typing, automated tests, broken-link-checking spiders, linters.

Mitigating Measures

You also have to start thinking about how to achieve Goal A with more than just preventative measures. Preventative measures aren’t everything. Even the most ardent block-testers have non-preventative measures: feedback from the end-user (even if it goes through 5 levels of support first). I’ll call these post-deployment measures “mitigating measures”. There are 3 main categories for the types of mitigation:

Reducing the severity of a defect (# of users, impact per user, etc)
Finding defects faster
Fixing defects faster

Hopefully those categories get your imagination started. User feedback is a mitigation measure; it’s just the slowest and most expensive one. Here are a few that do better:

Fast automated deployment. If you can’t get a fix out quickly, more users will be affected for longer than necessary.
Telemetry: logging, metrics, and alerts. Instrument your codebase in as many ways as possible to make it self-report issues it might be having. Alerts will tell you when things are going wrong faster than user-feedback and often you’ll learn about things that humans (manual testers or even end-users) did not, or even could not find.
One-click rollback. Make it so that your production environment can be rolled back to the previous known-good deployment as quickly and easily as possible.
Staged roll-out. Coupled with great telemetry, a gradual deployment process allows you to halt deployment and roll-back (even automatically) if there is a spike in crashes or logged errors before something is fully deployed.
Feature flags / Feature toggles / kill switches. Control (post-deployment) what users get what features and when. If there are any problems, only the allowed users are impacted, and the broken functionality is quickly reverted.

Of course, you will probably want to do these things even if you never eliminate block-testing from your release process. The prevailing practice seems to be to forego these things and expect the human block-tester will catch everything though. That’s a kind of bury-your-head-in-the-sand quality management.

If you try these things out though, you’re almost certainly going to see that these practices are cheaper, faster, and more effective than block-testers though.

“But Automation is Expensive!”

Automation IS expensive! I have no argument against that. It is. I’ve seen numerous well-covered codebases have more test code than application code, so the upfront costs of automated tests are probably close to the same as writing the application code itself.

Over the long haul it pays for itself though. A well-covered codebase easily has thousands of tests each getting run thousands of times a year, so we’re talking about millions of behaviour verifications on the conservative end of things. Humans simply cannot compete with that. Humans are slow, and even the best of them are terrible at repetitive detail-oriented tasks. As expensive as automation is, block-testers are even more expensive.

“But developers are terrible at testing!”

They’re really not. Many haven’t had much practice because they’re working in block-testing environments where the responsibility is muddled, but they’re actually quite good at it with a little practice when they have the full responsibility and trust for it. Don’t be surprised when they don’t choose manual-testing for the solution to every quality problem though.

“Do we fire our QA?”

Block-testing is so common in the industry that many people have a really hard time understanding the place of specific testing personnel in the SDLC without it. There's absolutely a place for specific testing personnel, but they've got to start contributing to improving quality beyond block-testing, and that change is understandably difficult.

Compared to computers, humans really are slow and terrible at repetitive detail-oriented tasks. However humans have well-known strengths that computers do not: they’re creative and curious.

So there are still places for manual testing:

Post-deployment exploratory testing. This is a great match for curious humans, and doesn’t slow delivery.
Feature-by-feature for new features only, with the developers present and actively involved, before code-complete of that feature. It’s probably best if the tester doesn’t actually do the tests themselves at all, but instead talks the developer through what tests to perform. This hands-on approach improves the developer’s testing ability more permanently and doesn’t confuse the fact that the developers own quality.

With that said, I’ve been working with multiple teams for the better part of a decade that have no manual tester role at all. When they do test manually, the developers are testing things themselves, often as a group (mob testing!). You’ll probably want to at least consider eliminating the role of manual testing entirely.

There’s still a place for quality-minded people other than manual testing though. In fact, the concept of QA and block-testing are really opposites. Actual quality assurance is about making sure that quality is baked into the process from beginning to end. Personnel that just do block-testing are not doing that at all. At best they’re quality control (QC) and that’s a far less valuable role. QA would be involved in a bunch of completely different concerns: How can we prevent defects from ever existing? How can we find them faster? How can we mitigate their impact? How can we recover from them faster?

Here’s a laundry list of possible Quality Assurance tasks:

run learning post mortems.
measure quality in many different ways (defect rate, MTTR, MTBF, etc).
regularly visit production logs and metrics to look for live quality issues
coach devs on how to more aggressively test their work
get into test automation
liaise with users and customer support about quality issues
help establish quality criteria for a task before it gets started, and throughout its development
Look for patterns in defects

“Does it really never make sense?”

The economics of block-testing make more sense if you anticipate very few releases with increasingly smaller differences. Agencies write this kind of software, but I haven’t personally done this kind of work in over a decade so I could be convinced of the economics either way by someone with more recent and extensive experience . Here I’m specifically talking about teams working on a software product that exists over a long period of time.

“But my situation is different because…”

Okay! I believe you! There are rarely one-size-fits-all practices in software development. I’m simply submitting this counter-argument for consideration. I’ve certainly worked at and heard from many organizations that should strongly consider stopping block-testing because the value proposition for them is just not there.

The Downward Spiral of Test-Later Automated Testing

2020-07-25T15:29:55Z

I’ve seen this pattern a number of times in my years as a software developer, and it’s always hard to watch:

(1) “We don’t need automated tests on this stuff. It’s way too simple.”

( The codebase grows until it’s more and more out of control. We’re getting scared to make changes and manual testing is taking forever. )

(2) “It turns out that we need tests, but this stuff isn’t very testable, so the most focused tests we can write are integration or e2e tests.”

( Test suite starts off well, but quickly becomes very slow and very flakey. When tests fail, it takes a lot of debugging to figure out why. They also require all kinds of complicated setup making them too hard to write.)

(3) “Automated testing is really expensive compared to the benefit! We’re probably not going to bother to write tests for simple things.”

It’s obvious why this pattern keeps occurring: Every next step makes sense given the current state.

It’s hard to break out of the cycle too, because it’s self-reinforcing. Working around the damage seems to create more damage. The things that seem like obvious solutions from an “in the spiral” point-of-view only make the problem worse.

Undoing the damage takes a lot of expertise and investment. But most importantly it’s expensive as hell the longer it goes on. You waste time:

avoiding necessary changes/improvements that are too scary
writing hard-to-write tests
debugging huge sweeping e2e tests
maintaining overly complex code (“You shouldn’t have to change the code just to test it!”)
doing more manual testing
rerunning flakey tests
investigating more obscure testing tools and strategies
waiting for test runs to finish
waiting for releases to be approved by humans
dealing with defects in areas “that aren’t worth the effort to test” or are “too simple to test”

Over the long run, cutting quality measures almost always costs you more than you gain.

The Trap of Solution-First Thinking

2019-02-24T16:34:04Z

I think that the most important thing I've learned while trying to improve my abilities as an organizational change agent (read: a programmer that likes to get his way), is to never ever present some idea as a best practice. If you don't know the reasoning well enough to explain it with that reasoning alone, you don't understand the problem or the practice well enough to suggest it.

Escaping Solution-first Thinking

One of the most powerful techniques you can use for convincing people is to stop coming to them with improvement ideas when the problems they solve aren't already obvious to them. I've been on both sides of these kinds of attempts to convince people and they rarely work. When I'm on the receiving end and I hear "Hey Gregg, I think we should do X", I immediately need to know:

What's the problem you're trying to solve?
Why are you prioritizing that problem ahead of our other problems?
What other solutions did you consider for solving that problem?
What are the pros and cons of each possible solution?
What made you consider this solution to be the best of all them for your situation?

The first question is the foundation of it all though: What's the problem you're trying to solve? When I'm thinking of suggesting a practice and I can't immediately and clearly answer that, I know I'm stuck in solution-first thinking. No one will be convinced by someone with that approach except the very few that happen to share your biases. I generally drop any idea like that entirely at that point instead of inventing an after-the-fact rationale. If my idea wasn't conceived "problem-first", it's not coming from the right place.

(I once had a junior engineer bravely ask "Why do we use version control?". I had been using version control for almost 2 decades and still really needed to force myself through those 5 questions. It was illuminating for both of us.)

"Best Practices" imply a Fixed Mindset

Is your best practice really "best"? Will there never be a better one? Did you give your team a chance to find their own solution that is better for their particular situation?

In general if you think you've found a best practice, you're probably not going to look for one that's better. You'll be standing in the way of your team improving and in fact the craft of software development improving as a whole. You'll be the last to adopt a better a practice if someone presents one.

What you call a "best practice" today came from an open-minded team that decided they wanted to try for something better than the current state of practice. It didn't come from a team that was happy with the status quo.

There's Value in Emulation

With all of that said, for things that are inconsequential, just using a "best practice" is often the smartest thing you can do. One real-life example for me was when some engineers on my team decided to use http's DELETE method to update a record in a new way after already using PUT for the usual scenario. They tried it locally and it worked. Why they didn't simply just use a POST to a new url was beyond me, but the reinvention of that proverbial wheel came back to bite us when our reverse proxy didn't allow bodies on DELETE requests. The lesson here is that if there's a standard way to do something and you're working in an area that doesn't really matter to your team/company, I'd always recommend just using the standard way. Sometimes you do just need a proverbial wheel and there's no reason to reinvent it.

Emulating others is a necessary step in learning too. If you're used to looking at practices used elsewhere and considering reapplying what has worked, definitely keep doing that. You should always keep learning about practices used elsewhere and it would be wasteful to always be reinventing the wheel by working only from first principles.

What I'm saying though is that for practices that really matter to your team, the next level is to additionally take the time to deeply understand:

What problem is being solved by that practice
At least a few other possible solutions
The downsides of that practice (there are almost always downsides)
If the practice is still the best one amoung all the others for your situation, and why

Being rigourous in your understanding of a practice and its rationale ensures that you don't apply the practice unnecessarily, or to the wrong problem, or incorrectly. Without this, you're just emulating a mature team and not actually maturing; you're cargo-culting.

Proxy Goals as Best Practices

The anti-pattern of "best practices" promotion includes arguing for hand-wavey proxy goals too. I'm talking about reasoning like "Let's do X because X is..."

RESTful
functional
object-oriented
what Linus Torvalds does
what I did before on my previous team
DRY
Agile
Lean
trending on hacker news
what jira expects us to do
what github expects us to do
what the scrum guide expects us to do
what facebook/spotify/[insert famous company here] does
what a great blog post I just read says to do (I'm pretending there's no irony here)

You may think some or all of these are great proxy goals to pursue (and maybe some are) but I personally think a few of them are terrible proxy goals for almost any situation and I can gaurantee that other people do too (some of which are probably on your team). Ultimately there are two problems with arguing for proxy goals:

(1) Proxy goals can become misaligned with your actual goals. (2) When you're trying to convince someone, you very rarely will know if they are aligned with your proxy goals. They're much more likely to be aligned with the team's or the company's actual goals though.

I've seen examples of all of these types of arguments for proxy goals (and made a couple of them myself) and though they can feel like shortcuts to a convincing argument, they really just make your job as a convincer much harder. First you have to convince somebody that Agile, for example, is a good goal, and then you have to convince them that your solution is indeed the most Agile. Usually they're just trying to get work done, so these unnecessary leaps of logic are extremely taxing on them. You're not going to win the hearts and minds this way. I've been through a few Agile transformations now, and the only successful one was the one where I never used the word "Agile" (It turns out that in general, jargon alienates people! Who would have thought?). Suggest real solutions to real problems in plain English and people will get behind you.

Results are what matter

Certainly your customer and/or the company's stakeholders generally won't care about what you think is a best practice if it's not obviously changing actual results. I've personally seen attempts at many of those indirect goals I listed be both successful to one team's goals and detrimental to another's. When they're detrimental, you've blown all your social capital for future influence, so it's important to get them right. And any practice is hard to get right if you don't understand why.

Throwing out your "best practice" mindset and stopping solution-first thinking forces you to be working on solving actual problems. Solving actual problems gives you a lot of clout within your organization, and allows you to more easily push for more ideas in the future.

A Javascript to Typescript Conversion Case Study

2018-12-11T15:01:43Z

Evergreen Update ( 2022-01-35 ): After some time working with Typescript, I'm actually completely sold on its usefulness and value, which I certainly was not at the time of this writing. I was surprised that none of the usual explanations seemed really true, eg it didn't reduce the number of tests or types of test that I wrote at all. What it did do though is it greatly sped up my feedback loop: (1) Things that I used to run tests for (even subsecond tests) were instead immediately apparent right in the IDE. (2) Things that I used to read tests for (like method signatures and object shapes) were instead immediately apparent right in the IDE. I wouldn't say my quality improved, or that my teams can manage larger codebases; I'd just say that we're all faster now at doing the same things. Also, I'm pretty sure that if you're not doing heavy automated testing, none of this will ring true for you.

And now for the original post:

I'm part of a team that's just finishing up a Javascript-to-Typescript conversion of an approximately 60 K LOC (lines of code) Javascript web service (resulting in about 86 K LOC of Typescript), so I wanted to throw some anecdata at the conversation around the costs and benefits of static-typing.

A few caveats worth mentioning right away:

We continued adding features while converting.
We spent a bunch of time additionally working on making the service's input types definable in Typescript with runtime validation (we convert typescript to json-schema during the compile step). This expanded the effort significantly, but it also allowed us to trust that type safety would hold at run-time.
We additionally converted from Promises to async/await as we went.

Alright, Let's do some math

In calendar time the conversion took about a year, but I estimate the actual developer effort was about 6 engineer-months. ++

85,000 LOC over 6 developer-months is around 14K LOC per month, or 475 LOC per day. From a naive line-by-line viewpoint that rate could seem slow, but there were many things that are easy in Javascript that are just infeasible in Typescript and had to be reworked entirely. +++ We also spent a chunk of time improving performance of compilation.

While I'm giving LOC stats, it's going to be worth noting that this codebase additionally has ~100K LOC worth of tests (that we left in Javascript), giving us a little over 80% coverage.

The Importance of our Test Suite

During the conversion, we found surprisingly few defects in the existing code. It was common for a single developer to be working on conversion full time and not find any defects at all in a week.

I probably converted over 10% of the code myself and never personally ran up against any defect that was user-facing. We ultimately did find a handful of user-facing defects in the pre-conversion code though (maybe 5?) and most had to do with the input validation not being strict enough. Typescript was useful for helping us find those cases in a way that tests probably wouldn't be (It's rare for our automated tests to be testing bad input extensively).

The conversion process did in fact introduce test breakages quite regularly though (that would have been bugs). We used the "noImplicitAny" compiler flag to help us enforce using types everywhere. At the file-level, this is an all-or-nothing conversion strategy, so on a large or complicated file, conversion could take hours before you could run the tests again, and continually I was finding that even with a successful build, the tests would fail in dozens of different unexpected ways, all indicating new defects.

Let me repeat that for emphasis: the act of changing existing working code to add types introduced defects that only the tests caught. This probably happened in more than half the files that I converted.

Conversely, what we've accomplished with tests (and no type-checking) on this codebase over the last 5 years is pretty mind-boggling to me. We converted:

From callbacks to promises
From express 2.0 to express 3.0 to koa.
From mongodb to Amazon Aurora for core data.
From coffeescript to javascript

And throughout each of these conversions, we deployed multiple times per day. None of the conversions were "big bang".

What I like (because I like Typescript):

Well I'm using VS Code now, instead of just vim, as one does when switching to Typescript. It's more sluggish of course, but it comes with better in-editor support for Typescript and that means that:

I can see a bunch of potential defects in-editor without running the tests at all. That's fast feedback.
Autocomplete is a lot smarter than what I've seen in any vim plugins.
Documentation about different function interfaces or object shapes is available right in the editor.

That last one is really key for me and ties into a larger improvement than just those in the IDE. Typescript encourages us toward standardizing object "shapes". With Typescript interfaces, we can say things like "Alright a 'user' will always look like this. It will always have these fields." If some other representation has less fields, or more fields, and we really need that, we'll have to face the pressure of adding an additional type. That's pretty powerful and I really appreciate it. Before, just reading application code, I'd have to consult tests to see exactly what shapes and interfaces were supported or returned and there was no pressure to simplify and consolidate on just one shape. That hurts learnability and maintainability.

There are of course a bunch of places where defining types is a productivity drag. It's certainly more code to write and more code to read, and for certain types of work, it can amount to just noise. When writing new code though, I do really appreciate how easily I can call on file-external classes/methods/functions/properties without having to jump around from file-to-file and figure out the proper way to do that.

Static Types and Quality

It's pretty clear to me that we're not really catching many more defects with Typescript compared to the effort that we've put into it.

We're catching defects faster though with faster feedback, right? Well I don't know about that either. We definitely are for certain classes of bugs. If a function that adds 1 and 1 together returns the string "2", that will get caught much faster. There are definitely lots of typos and misspellings that are caught more quickly too. I'm enjoying that immensely and it's a real productivity boost.

It's not free though. Compile time is around 40 seconds for me (and worse for others). So if I want to actually test runtime behaviour I'd say I'm now at a disadvantage. Firing the server up and manually testing locally now takes almost a minute when it used to take less than 10 seconds. Even with incremental compilation and a warmed cache, there’s a ~10 second compile time, which makes doing TDD (my usual way of writing code) a lot slower because of the slower feedback cycle. So, many types of defects are faster to detect, but many types are also slower.++++

But I don't have to write as many tests, right? I have no idea why anyone would think static-typing replaces certain kinds of tests. I was never testing types. Does anyone ever do that? I still write exactly the same number and types of tests. Testing hasn't changed a bit.

If it sounds like you're working in a scenario that is close to ours (a multi-year SaaS product), my recommendation is that if you have to choose between static-typing and tests, choose tests. To forego tests in this type of codebase is not only lazy but lazy in a way that actually leads to more work. It's hard to watch developers avoid the tiny upfront investment of tests when I can see what it's given us over the lifetime of this codebase.

You don't have to choose either, so if you're sure you want a statically-typed codebase, feel free to do that too. Typescript is not going to be a silver bullet though. The if-it-compiles-it-works approach is a recipe for long-term pain.

Additionally, if you don't have tests and you want to do a conversion from Javascript to Typescript on a large codebase, I'm fairly certain that you will go much slower than we did, and you will have a tonne of bugs. Of course our conversion is just one data-point, so your mileage may vary.

^endnotes
^{+ ...so it's not strictly true that 60 K LOC of Javascript converts to 85 K LOC of Typescript. I'm sure there are better sources of information for how many LOC conversion adds.}
^{++ I'm pretty confident in that estimate because around 80% of the conversion was done by one engineer fully tasked with the effort.}
^{+++ eg Most middleware stacks in node.js web frameworks rely heavily on monkey-patching which does not easily translate to Typescript. Also we were doing a bunch of sequential mutations of objects in Javascript that are really tedious to simply translate to Typescript because of the necessary proliferation of types.}
^{++++ I can't really totally blame Typescript for this. Our Typescript-to-JsonSchema converter is included in this compile stage, and it takes a significant chunk of the time. We really want to ensure that type-safety holds at run-time as well though. Additionally, I do think we should be relying on incremental compilation (with a watcher) more. It's possible with more effort we can fix most of the delay.}

Dynamic-Typing for Static-Typists

2018-11-05T17:54:37Z

I've always worked most heavily with dynamically-typed languages. They're a great choice for me and the teams that I've worked on because of the kinds of work that we're doing. It's pretty normal though for these teams to add team-members that are completely unaccustomed to working with dynamically-typed languages because they come from backgrounds where they worked mainly in statically-typed languages.

There are a few things about working with dynamically-typed languages that generally seem to blow the minds of new-comers that are accustomed to statically-typed languages, and I'll talk about a bunch of those things here. There's a huge amount of hype these days around type-systems too, and so I'd like to dispel any connotations that they're necessary in all cases, or that they're sufficient in any situation.

Get a Linter

Just because you're disconnected from static types, doesn't mean there aren't static analysis tools. Every dynamic language I know has a linter that can be integrated with most common editors or IDEs, and the linters are quite good at telling you in real-time about a bunch of common typos and mistakes you can be making. Make them part of your CI build too, instead of the compile step you're used to in order to keep them out of your codebase.

Write a Tonne of Tests

I know a bunch of people in statically-typed languages already write a tonne of tests, but if you're one of those ones that thinks that the static type-checker negates the needs for tests, you're reeeeeeeeally going to have to get over that and start writing some tests if you're going to be successful with dynamically-typed languages (and it'd almost certainly improve your statically-typed codebases as well).

It's pretty normal to write more test code than application code in dynamically-typed systems. There are no quality efforts that I've ever seen that have a higher pay-off than an extensive test suite though.

I've sometimes heard the argument that static-typing can mitigate the need for a whole tonne of test-writing but I've never found this to be remotely true. If you're testing output types and not the actual output values, you're doing it wrong. Just test the values; it's much easier and much more valuable.

I'll be brutally honest with you: If you find yourself on your first project that uses a dynamically-typed language and it doesn't have an extensive test-suite, you're almost certainly in for a very rough time as the codebase or the team gets larger. I generally have the same opinion about statically-typed codebases as well though.

Don't try to validate types at runtime on every internal method or function.

Of course you need runtime "type" validations. Just like in statically-typed codebases, you still need to verify user input, or that your data sources or 3rd-party services are giving you data in the format that you expect. You should absolutely do these validations that I'm going to call "edge validations" because they're on the boundary between your application and the outside world.

With that said, there's usually no good reason to sprinkle input-type validations on every internal function and method to try to replicate what you've had in the past with a statically-typed codebase. I agree that it's nice to assure quality internally (and I heavily use micro-tests for that), but 90% of the time a function will throw appropriately when it gets an argument type that it doesn't expect as soon as it tries to use it. Dynamically-typed languages employ Duck-typing, so that problems with input arguments are detected when and only when a function or method tries to interact with the argument in a way that's not expected. It happens at runtime, and the stack comes from somewhere deeper in function, but it's generally got all the information you need to debug it. Of course on its own debugging a problem like this would be much easier in a statically-typed language, but you've got a huge suite of tests not only to mitigate the problem of run-time-only detection but also to add validation of the actual values.

And with that said, I'll still sometimes (though very seldomly) add some assertions at the top of a function or method to ensure that the input arguments are valid. Often it's to check things about the input that most type-systems aren't much help with either, eg: "Is the number a positive one?" "Does the string start with 'http://'?". etc, etc.

Use the tests as (one form) of documentation

I've always loved the way that statically-typed systems enforce a certain type of code documentation. The type annotations make it easier for me to understand how to consume the functions and methods in the codebase and also what to expect inside the function or method if I'm making internal changes. The great thing about the type annotations too is that they're gauranteed by the type-checker to be correct whereas regular old source-code comments seem to always quickly go out of date (not that you shouldn't still try to use comments appropriately).

I'm probably going to sound like a broken record here, but again the extensive test suite can be quite useful for this. A test shows you how to call those functions and methods and what they'll return. I often find the tests more valuable than types too when I have both because the tests will show me argument and return values and not just the types (ie, It's more valuable to know that add1(1) returns 2 and not just a number type.)

Microtesting is specifically useful here too, because if you only do large black-box-style tests, your tests won't be very useful for internal documentation. It's also important to organize your tests in such a way that the relevant ones are very easy to find. For a microtest suite, you'll probably want to mirror your application file structure as much as possible. I'll certainly admit that statically-typed application code is better at putting that documentation right in your face as you're reading source code and that's really powerful, so at least try to make sure relevant tests are as easy to find as possible. (I haven't got any experience with Python's doctest, but it sure looks cool too.)

But most of all, when looking for valuable sources of code documentation, get used to considering tests and try to read them when you have questions.

Get the Right Attitude

Sometimes the most important thing that's required is an attitude adjustment. You're working with a different programming paradigm; you're going to have to do some things differently.

The fact of the matter is that static types are not necessary to...

...build certain types of software, and in some cases they're even a poor choice.
...avoid defects; they're just one possible tool, and not always the best one. (Try more automated testing!)
...build large systems or work with a large team. There are in fact many large successful companies built largely on dynamically-typed languages, and there are many large open-source software projects that build on dynamically-typed languages.

These dynamic languages have accomplished a hell of a lot in computing. You shouldn't need much more evidence than that.

Effective Technical Improvement

2018-10-14T20:23:13Z

One of my favourite ways to tackle tech debt is to fix it as I work through product requirements. There are some great advantages to it:

It automatically prioritizes tech debt burndown in areas that you’re probably going to touch again soon, which is where it’s most valuable.
It doesn’t require blocking any product development flow. It just slows it down (probably imperceptibly).
It doesn’t even require conversations outside of engineering.

I'd hazard to say this is probably considered a best-practice, so people making technical improvements this way are in good company. I call this "the organic method" because improvements are happening naturally, or "organically" as other changes occur.

There are some downsides though (especially with large codebases with many affected developers):

It hides the actual cost. Is it really better for product development to be imperceptibly slower? Wouldn’t it be nicer if costs were more explicit and obvious?
It’s a lot easier to do tech improvement and product development separately. Doing two things at once is almost always more complicated.
It’s easier to find patterns, better solutions and shortcuts for even fairly mechanical technical improvement work if you’re focusing only on that technical improvement work for some period of time.

Usually the biggest downside is that it’s slower.

In practice, I always find that it’s much much slower than you’d think. Here's a graph of a Javascript to Typescript conversion effort that I've been tracking for the past 11 months:

There are 2 small steep declines here that show the efforts of single individuals for short periods of time, but otherwise this graph (spanning almost a year) is a 64-file improvement out of 223 files in 11 months. At that rate, the effort will take 3.5 years.

I've tracked a number of similar efforts over the last year and the results are similar. My previous experience with organic improvement in large codebases feels pretty similar too: Without specific mechanisms to keep the conversion going, it naturally slows (or worse, stops).

Why does it matter if it’s slower?

Maintaining the incomplete state of the conversion is sneakily expensive:

It's harder for newcomers to learn the right patterns when multiple exist
Engineers need to remember all the ongoing efforts that are underway and always be vigilant in their work and in their code reviews of others work
Diffs are easier to understand when they don’t try to do too many things at once
Copy/pasta programming, forgetfulness, and uneducated newcomers lead to contagiousness; propagation of the wrong pattern instead of the right pattern
When you’re really slow, you’re even more likely to have multiple of these efforts underway at once, compounding the complexity of fixing them
If you’re slow enough and not actually doing cost-benefit analysis, patterns can be found that are “even better” than the ones underway. This is how you end up with even more ways to do the same thing and a smaller number of engineers that find joy in working in that codebase.

Most importantly though, if there’s really value in paying for that technical improvement, why not pay it sooner rather than later? Ironically, most of the least productive (and least fun) codebases I’ve seen are because of people making numerous actual improvements but then leaving them only partially applied. Good intentions without successful follow-through can easily make the code worse.

For larger technical improvements (ones that affect too many files to pull off in a week or less) you want to make sure that:

You have a vague idea of the cost and you’re actually making an improvement that you think will be worth the cost.
The timing for doing it now is right (and there isn’t something higher value you could do instead)
You actually have a plan that converges on total conversion in a reasonable amount of time instead of something that just leaves the codebase in an inconsistent state for an extended period of time.
The goal, the timing and the plan are generally approved by your teammates (even if unanimity is impossible)

Once you've got those 4 factors in place, you're probably better off in the long run if you capitalize on the improvement as quickly as possible. You probably don't want to cease all product development for a huge amount of time to do it, or send one developer hero off to fix it all, but you'll probably want to come up with something better than organic improvement too, if you really care about that improvement.

Cross-functionality is a Secret Weapon for Product Development Teams

2018-10-03T22:14:59Z

In my experience, cross-functional teams align people to business goals best, and so they can get to real results much faster and much easier than teams made up of a single function. They really don't seem to be that popular, so I thought I'd talk about them a bit.

Here's some common chatter across mono-functional teams:

The Engineering team:

"We should never be taking on tech debt. Tech debt slows us down!"
"We should stop everything and clean up all our tech debt, regardless of cost or current business goals"
"We should convert all our code to use this new framework everywhere because it has [INSERT TODAY'S LATEST DEVELOPMENT FAD]"
"It's the testers' job to test, not mine"
"Works on my machine!"
"Let ops know that I used the latest version of the database client library, so they'll have to upgrade all the databases"

The Testing team:

"Let me see if I can find a reason to stop this release"
"We need X more days to test before the release"

The Frontend/Mobile/Ios/Android team:

"That bug is on the backend."

The Backend Team

"That bug is on the frontend."

The Operations Team

"We haven't got time to do the release today. Let's schedule something for early next week."
"Engineering doesn't need access to that. Just tell us what you need."

The Design Team

"We don't want to help with a quick and dirty design for that feature experiment. It doesn't fit into our vision"
"We've got the new total redesign for this month specced out."

The Product Management Team

"That technical debt burndown can wait, right?"
"We should do this the fastest way possible."
"Here are the detailed specs of what I want you to build. Don't worry about what problem we're trying to solve."
"I've finally finished our detailed roadmap for the year."

Do you see the patterns?

These teams...

optimize for the areas of their specialization, not for the business' goals or for other teams' goals.
defend their area of specialization by hoarding power and information
constantly try to expand their area of specialization at the expense of the business' goals
focus more on looking busy than getting real business results
push blame to others
too willingly take on expensive projects where others pay the majority of the costs or where the value isn't aligned with company goals.

So what to do instead?

Well you get these mono-functional teams because someone talking about a specialty or discipline once said something like "X is important. We should have a team for X."

My suggestion instead is simply to start saying "X is important. We should have X on every team."

This leads to a team with a bunch of different but cooperating specialties. The only thing they all have in common is their team's portion of the business' goals.

Think of it this way:

If the members of a team don't share their goals can they really even be called a team?
Why would you give goals to a team without also empowering them with all the specialist skillsets and ability to also deliver on those goals?

In general I've found that only a cross-functional team can make the proper trade-offs on its own, react quickly to changes in the world, and execute with minimal communication overhead. Once it has all the specialties it needs to autonomously deliver on its goals, you're set up for a whole new level of speed of execution.

I'm not saying that cross-functional teams solve all the issues above, but they make the conversations happen on-team where it's much cheaper than across teams, and the conversations are much easier because people don't have to guess each other's motives nearly as much.

It's not any easy transition either if you're currently on mono-functional teams. In my experience though, cross-functional teams can really make mono-functional teams look like like a morass of endless disagreements and easily avoidable meetings.

No Heroes in Tech Debt

2018-09-20T13:04:35Z

Too often when I see a team trying to replace a bad/old/deprecated pattern that is widespread in a codebase, they default to what I call The Hero Solution: One person on the team goes through and fixes every case of it themselves.

This can work for very small efforts, but it's almost always a terrible solution in larger efforts for a few reasons:

When the bad pattern is widespread this is the slowest way to fix it and the slowest way to get to value from the new pattern.
There's nothing in this policy that stops other people from continuing to add the bad pattern. Indeed there will often be code with the bad pattern that they want to copy/paste/modify, making the bad pattern almost contagious.
Teammates may be working in the same areas of the codebase causing merge conflicts that slow both the teammate and the hero down.

Here are a few tips that will see better results:

Track the bad pattern

Find a way to track instances of that bad pattern over time. Often a simple git grep "whatever" | wc -l will tell you how many cases of it you have in the codebase. Check often and record the values. Whatever your strategy is, if it's not trending toward 0 in a reasonable timeframe, your strategy is not good enough. Come up with something else.

I can't tell you how many cases I've seen of efforts trending toward multiple years (sometimes as much as 10 years) as soon as I started measuring over time, determining the rate of change, and extrapolating the completion date.

If you do nothing else, do this. You'll quickly be able to see the cost (in time) and be able to reassess the value propsition.

Stop the spreading!

Agree with the team on a policy that no new instances of the bad pattern will be added without a team-wide discussion. Add pre-commit hooks that look for new cases (git grep is awesome again) and reject the commit. Look for new cases in Pull Requests. Get creative! Without this, you're basically bailing a leaking boat without patching the leak and you will have:

People that want to copy/paste/modify some existing code that has the bad pattern
People that don't even know the pattern is now considered bad
People that knowingly implement the bad pattern because they don't know about the good pattern.

If you can't get your team mates on board, your effort is doomed. I've seen bad patterns actually trend upwards toward infinity when no efforts have been taken to get consensus or stop the bad pattern.

NB: Because of the regular human effort involved in maintaining consensus and educating and re-educating people about the goals, one of the best ways to stop the spreading is to concentrate on faster total conversion to the new pattern. Having bad examples all over your codebase works against you on a daily basis. Bad patterns are contagious.

Get your team mates involved in the conversion!

Here are a few ideas:

Have a rule where no modified files (or functions or modules or whatever doesn't feel too aggressive for your team) can contain the bad pattern anymore. Figure out ways to enforce this automatically if possible, or in code reviews if not.
Break the work into chunks and schedule those pieces within other product work on a regular basis. This is sort of a nuclear option, but if the chunks are small enough (yet still encompass the entire scope of the conversion), you can show regular and reliable progress without stopping production work for any extended period of time.
Get other people to help with the conversion! If people are bought into it, there's no reason one person should be doing it alone. Multiple people working on it (in a coordinated fashion) will reduce merge conflicts with product work, and increase knowledge sharing about the proper pattern. You may even get better ideas about how to convert.

Don't do things that don't work.

Stuff that doesn't work:

Efforts that the team as a whole doesn't find valuable / worth the cost.
Efforts that are ill-timed. Should you really do it now? Is this really the most important thing?
Efforts that are not tracking toward 0 in a reasonable amount of time. Partial conversions are really hard to manage. They may not be a strictly technical concern, but they are a concern for on-boarding, managing, complexity/mental-overhead, knowledge-sharing, etc. Come up with a strategy that doesn't prolong them.

Big problems need smarter solutions!

The Mathematics Behind Experience

2018-07-15T23:41:53Z

I always try to think about what exactly it is about experience that makes a software developer better. Are there aspects of experience that are teachable, but we don't understand them yet well enough to teach them? Can we do better to pass on "experience" rather than have every developer suffer through the same mistakes that we suffered through?

I know I've had a couple of hard-won lessons over the years that really helped me be more successful in software engineering. I've been able to see the warning signs of mistakes to avoid for a long time, but I think only recently I figured out the reasons behind those warning signs. And I think some of the reasons can be explained mostly in terms of probability math, which can then be taught right?

Before I go into this, I'd like to preface this by saying I've failed many many math classes. I hardly ever use any advanced math in my work, and doubt many other programmers do. I wish I had more patience and learned more math (especially how to apply it) too. So with that said, here goes...

Lesson: The Certainty of Possibility at Scale or over Time

The first piece of experience I'd like to pass on is that your "flawless" solution to a problem, at scale, over time, will fail.

I don't know how many times I've heard an engineer say something like "This solution is basically bulletproof. There's only a 1 in a million chance that that corner case you just mentioned will occur", and then promptly put the solution into an environment that does 1 billion transactions a month.

Here's how the math looks:

1B transactions/month * 1/1M probability of failure per transaction
= 1B / 1M failures per month
= 1000 failures per month.

Yes, the mathematics of probability are telling us that that particular solution will fail roughly 1000 times a month. This is probably a "duh" conclusion for anyone with a computer science degree, but I see developers (yes even ones with computer science degrees) failing to apply this to their work all the time.

At scale and over time, pretty much everything converges on failure. My favourite thing to tell people is that "At this scale, our users could find themselves in an if (false) { statement.".

So what does this mean? Well it doesn't mean everything you do has to be perfect. The downsides of failure could be extremely small or completely acceptable (In fact actually pursuing perfection can get really expensive really quickly). What this tells you though is how often to expect failure. For any successful piece of software, you should expect it often. Failure is not an exceptional case. You have to build observable systems to make sure you know about the failures. You have to build systems that are resilient to failure.

Often I hear people talking about solution designs that have high downsides in the case of failure with no back-up plan and defending them with "But what could go wrong?". This is a tough one for an experienced developer to answer, because having experience doesn't mean that you can see the future. In this case all the experienced developer knows is that something will go wrong. Indeed when I'm in these conversations, I can sometimes even find one or two things that can go wrong that the other developer hadn't considered and their reaction is usually something like "Yeah, that's true, but now that you mentioned those cases I can solve for them. I guess we're bulletproof now, right?".

The tough thing about unknown unknowns is that they're unknown. You're never bulletproof. Design better solutions by expecting failure.

Understanding this lesson is where ideas like chaos monkey, crash-only architecture, and blameless post mortems come from. You can learn it from the math above, or you can learn it the hard way like I did.

Lesson: Failure Probability Compounds

Here's the second piece of mathematically-based wisdom that I also learned the hard way instead: If you have a system with multiple parts relying on one another (basically the definition of a system, and any computer program written ever), then the failure rate of the system is a multiple of the failure rates of the individual components.

Here's an almost believable example: Let's pretend that you've just released a mobile app and you're seeing a 0.5% crash rate (let's be wildly unrealistic for simplicitly and pretend that all the bugs manifest as crashes). That means you're 99.5% problem-free right? Well what if I told you the backend has an error rate of 99.5% too and it's either not reporting them properly or your mobile app is not checking for those error scenarios properly?

Probability says that you compute the total probability of error by multiplying the two probabilities, ie:

99.5% X 99.5% = 99%

What if you're running that backend on an isp and that's got a 99.5% uptime? And your load balancer has a 99.5% success rate? And the user's wifi has 99.5% uptime? And your database has a 99.5% success rate?

99.5% X 99.5% X 99.5% X 99.5% X 99.5% X 99.5% = 97%

Now you've got a 3% error rate! 3 out of every 100 requests fails now. If one of your users makes 50 requests in their session, there's a good chance that at least one of them will fail. You had all these 99's and still an unhappy user because you've got multiple components and their combined rate of error is the product of each component's error rate.

This is why it's so important when ISPs and Platforms-as-a-service talk about uptime with "5 nines" or 99.999%. They know they're just one part of your entire system, and every component you have in addition has a failure rate that compounds with their baseline failure rate. Your user doesn't care about the success rate of the components of the system -- the user cares about the success rate of the system as a whole.

If there's anything at all that you should take from this, it's that the more parts your system has, the harder it is to keep a high rate of success. My experience bears this out too; I don't know how many times I've simplified a system (by removing superfluous components) only to see the error rate reduce for free with no specific additional effort.

Lesson: When faced with high uncertainty and/or few data, the past is the best predictor of the future.

We saw in the last lesson one example of how surprising complexity can be. The inexperienced developer is generally unaware of that and will naively try to predict the future based on their best understanding of a system, but the stuff we work on is often complex and so frequently defies prediction. Often it's just better to use past data to predict the future instead of trying to reason about it.

Let's say you're working on a really tough bug that only reproduces in production. You've made 3 attempts (and deployments) to fix it, all of which you thought would work, but either didn't or just resulted in a new issue. You could think on your next attempt that you've absolutely got it this time and give yourself a 100% chance of success like you did the other 3 times. Or you could call into question your command of the problem entirely and assume you've got a 50/50 chance, because there are two possible outcomes; success or heartache. I personally treat high complexity situations like low data situations though. If we treat the situation like we don't have enough information to answer the problem (and we've got 3 failed attempts proving that that's the case), we can use Bayesian inference to get a more realistic probability. Bayesian inference, and more specifically, Laplace's Law tells us that we should consider the past. Laplace's Law would say that since we've had 0 successes so far in 3 attempts, the probability for the next deployment to be a success is:

  = (successes + 1) / (attempts + 2)
  = (0 + 1) / (3 + 2)
  = 1 / 5
  = 20 %

This is the statistical approach I use to predict too. With no additional information, if I've failed at something a number of times, the chances of me succeeding in the future are reduced. I don't use this data in a depressingly fatalistic way though -- I use it to tell myself when it's time to make more drastic changes in my approach. Something with a 20% chance of success probably needs a much different, more powerful approach. I also stop telling people "I'm sure I've definitely got it this time".

Similarly, if there's an area of the codebase that has received a comparatively large number of bug-fixes, I'm going to lean towards it being more likely to have bugs than the areas having less bugfixes. This may now seem obvious, but if you're the person that did all those bugfixes, you may be biased towards believing that after all those fixes, it must be less-likely to still be buggy. I'd happily bet even money against that, and I've actually won a lot of money that way.

I think it's fair to say that this is extremely counter-intuitive and may on its face look like a form of the Gambler's fallacy, but remember it's for use in low-information scenarios, and I would say that numerous defects are clear evidence you didn't have enough information about the system to predict this bug in the first place.

Relatedly, Extreme Programming has a simplified statistical method of estimating how much work a team can handle in a sprint called "Yesterday's Weather". Instead of having a huge complicated formula for how much work a team can handle in a given timeframe, they simply look at what the team was able to accomplish last sprint. It's going to be wrong a lot of course, but so is whatever huge complicated formula you devise.

If there's a more generalized lesson that you should take from this, it's that we work with complex systems, they're notoriously difficult for humans to predict, and predicting statistically can often get you a clearer picture of the reality of a situation. Resist prediction from first principles or through reasoning. You don't have enough data, and software systems are normally elusively complex.

With all this said, I know it's pretty hard to really get a feel for the impact of probabilities until you've had the prerequisite defects, failed deployments, and product outages. I sure wish someone had at least taken a shot on telling me in advance though. It also makes me wonder what I've still got left to learn.

Zero Defect Policy

2018-03-28T00:32:11Z

The 4th criteria of the Joel Test of quality on a software team is:

Do you have a bug database?

You probably do. The test was written almost 20 years ago and if I recall correctly almost everyone had one anyway. Joel was even selling one.

On the surface it seems like common sense. Quality is something you want to manage. So naturally you'll also tend to want to log things... measure things... track things.

I want to propose a better way for most situations though. It's simpler, your long-term speed will be faster, and your product quality will be higher:

The Zero Defect Policy

Simply put: Prioritize every defect above feature work or close the issue as a WONTFIX. I'm not suggesting you interrupt any work already in progress, but once that's done, burn your defect list down to 0.

Why in the world?

Bugs are much cheaper to fix immediately. You know the product requirements most clearly at that time. You were touching that code most recently at that time and it's still fresh in your mind. You might even remember a related commit that could have been the cause. The person who wrote it is still on the team.
Most things we'd classify as bugs are really some of the most obvious improvements you could make. You probably don't need to a/b test a typo fix. If your android app is crashing on the latest Samsung phone, you don't need a focus group.
Managing bugs is a huge effort. If you're not going to immediately fix them, you have to tag them, categorize them, prioritize them, deduplicate them, work around them, do preliminary investigations on them, revisit them, and have meetings about them.
The development team needs the proper immediate feedback and back-pressure to know when to speed up and when to slow down.

Say What now?

Defects are perfectly normal. If you don't have them, you're either NASA or you're developing too slowly. However, if you have too many, or if they're particularly costly, you absolutely need to slow down and turn them into learning opportunities. In those times, the fix isn't enough. The fix with an automated test isn't even enough. You'll want to look into:

other prevention measures
faster detection measures
faster ways to fix
better ways to limit impact

This is the REAL definition of Quality Assurance. If you do this thoughtfully, and don't try to make an army of manual testers the best solution you could come up with, over the long term you'll be much much faster. You'll be the kind of product development team that the company actually believes when you say something is "done".

What about the low value, high cost bugs?

Delete them. If they become worthwhile later, you'll hear about them again. If you can't bring yourself to delete it, you probably value it too much to not fix it. Just fix it.

What about when my team ends up with a week's worth of defects and can't get any feature work through?

There will definitely be dark times. Slow down and learn from them. It's the best time to talk about speed improvements, because speed and quality are interdependent. You can move much faster in mistake-proof environments. In most cases, nobody cares how fast you ship broken software.

Sounds great, but what about our existing bug database of hundreds of bugs?

Many are probably not even bugs anymore if you've been collecting long enough to get hundreds. How many of those old things can you even reproduce? How many have clear enough descriptions that you can even still understand the problem? Is the original submitter still around? Here's my solution: delete all but the highest priority ones and immediately schedule the high priority ones above other features. Allow people to file reports again if they really care about a particular defect and the defect still exists.

The bug database is great in theory, but in practice, it's often an aging garbage heap of ignored customer frustrations and a /dev/null for opportunities for improvement.

Enterprise Iffing 3.0 for 10x Rockstar Programmers

2017-10-27T14:57:02Z

I've had a chance to get back to coding after a couple of months' hiatus so I thought I'd write about something a bit more fun and way down in the details.

Conditionals add complexity -- they seem like a tiny bit of complexity in the singular case, but they really do add up quickly to make code unpredictable and difficult to follow.

Your debugger and your unit tests are probably great at managing that complexity, but they're really kind of a crutch. You want to remove complexity, not manage it.

As a result I have a bunch of rules around conditionals that I generally always try to follow.

Early-Return is Simpler

Probably my most important personal rule is to use early-returns wherever possible. In general, it's not simpler to just have a single return at the bottom of a function when you can return earlier in some circumstances. If you know a value shouldn't be changed after a certain point in a function you should just return right there. There are two reasons it's simpler:

You don't have to try to reason about what might still happen later
Subsequent reading of the code (by you or others) can be much faster because you can stop reading the code for a function as soon as the case you're investigating hits a return.

function fizzMyBuzz(i) {
  var output;
  if (i % 15 == 0) {
    output = "FizzBuzz";
  } else if (i % 3 == 0) {
    output = "Fizz";
  } else if (i % 5 == 0) {
    output = "Buzz";
  } else {
    output = i;
  }
  return output;
}

function fizzMyBuzz(i) {
  if (i % 15 == 0) {
    return "FizzBuzz";
  }
  if (i % 3 == 0) {
    return "Fizz";
  }
  if (i % 5 == 0) {
    return "Buzz";
  }
  return i;
}

There's a visible difference in density here. The second example just has less stuff, even though the number of lines are pretty similar. Early returns mean that subsequent code has to worry a lot less about the effects of previous code.

There's definitely a trade-off here. Now when you're looking for where the function returns, it's not just at the end -- it could be in a bunch of places. I think the trade-off makes sense though because even when I'm tracing through function calls in a backward-fashion, I'm usually reading functions from top to bottom.

Try to NOT use negation

Negation is complexity. In my experience it probably even beats off-by-one errors in the category of "things that are too simple to possibly go wrong that go wrong all the time".

Example:

Don't allow additional vegetables in this salad but avoid forbidding the addition of non-orange vegetables

So can this salad have carrots or not?!?

No orange vegetables

Oh

Something like...

if (!specialCase){
  someStuff();
} else {
  someOtherStuff();
}

...should at the very least be changed to:

if (specialCase){
  someOtherStuff();
} else {
  someStuff();
}

One exception to this rule: If negation will let you early-return, definitely do that! Hopefully this non-non-negation exception is not too complex. ;)

Now you can pretty much deprecate `else`.

Often when I can't return early, I just move the entire if..else block to a method where I can return early. Then I don't need else.

Trying to restrict my use of else is a great forcing function for creating smaller functions and using early-return more often.

for (var i=1; i <= 20; i++){
  if (i % 15 == 0) {
    console.log("FizzBuzz");
  } else if (i % 3 == 0) {
    console.log("Fizz");
  } else if (i % 5 == 0) {
    console.log("Buzz");
  } else {
    console.log(i);
  }
}

for (var i=1; i <= 20; i++){
  console.log(fizzMyBuzz(i));  // We wrote this earlier in the post!
}

Working Without Estimates

2017-10-22T01:17:00Z

I've been developing software for a few years without doing much estimation at all, and the estimation I've been doing has been really vague, eg "You know that feature is a lot of work, right?". I've recently been reading more and more from the #noestimates movement on The Twitter, so I thought I'd chime in a bit with my rationale and experience as well.

"Is the company crazy?"

Yes, but I don't think the lack of estimation contributes to that. Estimation has a bunch of downsides to it that make eschewing it pretty rational:

Teaching people estimation best practices takes a lot of time. Without spending that time, the estimates are terrible. Story points, planning poker, and other methods seem to blow people's minds.
Estimation meetings take a lot of time.
It's almost impossible to ensure that people (engineers or management) are not treating the estimates as deadlines and unnecessarily rushing work (creating unnecessary defects and technical debt that slow the team down more).
A lot of estimates are still really terrible and therefore low-value and not worth the effort. Decades of industry-wide software development thinking have not changed that.
A lot of estimates don't actually matter. A 1-week task that is off by 50% but still successful with users, is almost always still a huge success. Sure being off by 50% on a 6 month plan is really bad, but my best recommendation for 6-month plans is "don't".

"But then how do people know when to stop working?"

You've probably heard Parkinson's Law that "work expands to fill the time available". In my opinion it's a super-cynical way of thinking about people but I've known many people that believe it. If it seems true to you in your organization consider these factors that might be contributing:

Engineers almost always under-estimate (unless they're consciously trying to under-promise/over-deliver -- cynically called "sand-bagging"). This is because they're estimating best-case scenarios, and life with complex socio-technical systems rarely falls in the best-case.
The focus on how long things take often incentivizes engineers to take short-cuts and to just allow technical debt and complexity to pile up. If they've got extra time in an estimate for some reason (hard to believe because of the previous point), they actually start doing the refactoring that they would otherwise not consider doing.
They may be gold-plating something because they don't know what the next most important thing to do is, they don't understand or believe in the value of it, or they don't think they'll get a chance to ever return to the current feature to tweak/improve it. In this case, there are probably trust issues to solve.

Instead...

Make sure people are adequately incentivized to move on to the next thing. Probably by making them care about the next thing.
Have a clear definition of what it means to be done (ala Scrum's Definition of Done).
Give them all the time in the world for the current thing. Let them relax and do it right so it stays done instead of being a constant source of future interruption as the defects roll in.

If you have trust and the right people, the team will move really fast, especially over the long haul.

"Estimates can be a good forcing function though. Otherwise how do people know when to cut functionality?"

Just cut it all immediately. Decide the absolute essentials of the feature, and cut the rest immediately with no mercy. Deliver those essentials first. Then incrementally try to fit in the next most valuable aspects.

This is the only sane way to ensure you're doing the highest value work first. You don't need scheduling -- you just need prioritization. The stuff that gets cut will be the low-priority stuff.

"I need some level of predictability in the software development process though!"

That doesn't mean you're going to get it. If you've been in software development for any length of time, you know that estimates are often wrong and you haven't figured out how to make them better. In complex systems, prediction rarely leads to predictability.

Instead of trying to predict, you should be aiming to try to mitigate risk as early and often as possible, by doing the riskiest, least clear, and highest value proposition efforts first in the leanest way possible.

This allows you to incrementally move to the more proven, clear, and next-highest-value efforts over time, and have an effort that observably converges on completion. It's a better shot at predictability, but without prediction.

"But everyone else does it!"

This is the worst of the arguments, in my opinion. I love employing best practices when I don't have a better way but otherwise they're the enemy continuous improvement. Estimates on their own deliver no user-value. Any effort that isn't yielding comparable results should be axed. Any goal that can be better served by other methods should just be solved by those methods.

If you're not a developer and you still don't buy my argument...

Let's flip the script.

As an engineer I've yet to meet anyone in management that is willing to give me estimates on the expected value of a feature (preferrably in dollars, but I'll take whatever the proxy/vanity metric of the day is too!). This would be super-valuable to ensure we're prioritizing the most impactful stuff first right? And we could check the validity of these estimates after the feature is released right?

I think this would be a hilarious way to turn the tables and see how management does with estimation of complex systems, but in the end I think it would be similarly fruitless for improving predictability. Their estimates would be just as wrong, just as often. They'd be just as nervous about being confronted on their accuracy too.

Estimation is a linear process management tool in a non-linear world.

Estimation just doesn't really provide us with much predictability considering it's cost.

That doesn't mean that no one should ever do it -- it will definitely make sense in some cases -- but I personally think it's overused, largely ineffective, and often destructive in most of the cases that it's used.

Why does Bad Waterfall Happen to Good People?

2017-07-20T22:35:31Z

I've recently seen a case of spontaneous waterfall-style development emerging where the team was generally empowered to avoid it or at least to correct against it, and certainly wanted to, and yet did not. From the outside it seemed like waterfall was a kind of steady-state for them that they'd trended toward and didn't have the escape-velocity to move toward anything else. They weren't happy with it either. Most of them felt quite alienated from their work which is a common result of waterfall development. What could have happened?

I think there are a bunch of properties that can lead to waterfall-style development and management is only one of them. Before I get into those, I'd first like to propose a simplified explanation of what it means to be waterfall.

What’s This Waterfall Thing That They’re Talking About?

Waterfall development is development with varying degrees of one-way hand-offs between multiple disciplines. The best way to think of it is to imagine a sort of human assembly line where each homogenous set of disciplinarians pass the result of their discipline down to the next homogenous set of disciplinarians. More concretely, you've got these hand-offs when the product managers tell the designers what to design and the designers tell the engineers what to build and the engineers give the testers the work to test and the testers give the release manager a release to release and the release manager gives the sysadmins a deployable to deploy. It's all very well organized and simple to explain and generally soul-sucking and totally inefficient in practice.

It's soul-sucking for a few reasons:

The downstream disciplines have very little power to make drastic changes, because of the tyranny of sunk-cost fallacy. For example, if the testers find some behaviour that is terrible for the users, the engineers will shrug and say "not a bug... works as designed!". Or if the engineers determine there's a performance problem with the fully completed hi-fi design mocks that make them infeasible, the designer is at best forced to go back to the drawing board to take another shot at getting an idea that the engineers might agree on.
The downstream disciplines have no idea why they're designing/building/testing/releasing what they're designing/building/testing/releasing. Waterfall is set up as an assembly line in the name of efficiency and that means that downstream disciplines are on a need-to-know basis. The problem with this is that you suck the motivation out of everyone because knowing "why" is precisely where motivation comes from.
The upstream disciplines are generally resented by the downstream disciplines for how they're always ignoring the important downstream concerns.
The upstream disciplines are constantly pressured to do more and more flawless work "to minimize all that expensive back-and-forth".

It's funny when people start to talk about "all that expensive back-and-forth" because trying to avoid that is precisely what makes it so expensive. The upstream disciplines are always expected to trend toward perfection in pursuit of the ideal one-way waterfall. When you hear the engineers asking for perfect hi-fi mocks before they start their work, you'll know you're approaching waterfall-land. Your product development flow is about to go into slow-motion.

The Role of Single-Discipline Teams

There's an idea too that similar disciplinarians should be the ones working and managed together as a unit; that's how you achieve consistency in the discipline and how you ensure these specialists are always pushing each other to greater heights in their respective crafts. Unfortunately these specialists never really get a full view of the entire process. With neither broad understanding nor a holistic ownership mindset, the end result is that you’re paying a whole lot of smart people while simultaneously trying to avoid them thinking about the business' problems. A specialist that doesn't fully understand the business goals or how software development works in general will always be a pretty weak specialist regardless of the depth of their specialization.

This is how you get dysfunctional scenarios like:

QA personnel that feel like their primary goal is to try to find a way to block the release.
Designers that feel like customers should wait for the perfect aesthetics before we can deliver new functionality.
Engineers that can’t make good decisions about when to take on technical debt and when to pay it down.

Having individuals confined solely to their area of expertise can make sense in the short term! For example, I'm an engineer and I've regularly had terrible product ideas. In the short term there's no sense in involving me further "upstream" in the process; I'll only drag it down.

In the long term though, my terrible product sense is certainly a liability. And on a daily basis when I stand in the shower or on the subway thinking about work, I'm just coming up with more terrible ideas. The better thing to do is to teach me about the business and its goals and make that investment in me. You'll occasionally get a worthwhile idea from me, and I'll always be more willing to go that extra mile because I know why I'm building what I'm building and that it's worthwhile work.

The Role of Product Management

Product management can play a significant role in whether or not that teaching and that context-sharing happens. I’ve seen teams prioritize their own product backlog for multiple weeks of work without a product manager present -- That’s a team that the product manager has really invested in and the result is that the product manager is comfortable giving them agency. There are some unexpected things that can happen here: the team will often have better ideas or prioritizations than the PM, or at the very least will have top-of-stream improvements on the PM’s ideas that would never happen otherwise. When this is really working, the one brain of the PM will almost certainly not be able to beat the multiple brains on the team. And when the PM considers themselves a contributing member of the team, the team will necessarily outperform the lone PM.

There’s a simple but slow progression that a skillful PM can push the team through if they’re in waterfall-mode due to the product management stage:

Allow the team to own the solutioning. Take the team from working on predetermined solutions and start to instead involve them in determining the solutions themselves. Achieving this really takes a considerable amount of effort, but it usually pays for itself in spades later with the PM’s freed-up time, better work prioritizations, better product decisions, and less process churn (“expensive back-and-forth”). It’s amazingly motivating for the team too.
Allow the team to own the problem-selection. Once you’re comfortable with their performance in determining the solutions, start to get their input on what the problems to solve are, and coach them to seek problems that meet the team’s goals and show them to independently collaborate with other sources of company/product/user information. Give them access to the metrics they’re trying to improve. In time, be open to letting them try some ideas that you might not think are great; at the very least it’ll be a learning.
Allow the team to have input into the mission. Everything is easier when the team thinks it’s working towards the right goals. The easiest way to get the team working on the right goals is to ensure that they have that context. If you brought the team through steps 1 and 2 above, you can see that they’re much more bought-in to plans that they’re part of. Involve them in actually determining the mission too! The end result will be a team that is incredibly fast-moving and energized.

Not to be too deep, but even though this is framed here as a maturity progression for the team, it can also be a maturity progression for the PM. The PM needs to be able to become the team’s guide to the market, the user’s needs, the company’s goals, etc, etc, instead of playing the part of puppeteer. And we’ve seen that it absolutely doesn’t put the PM out of a job. There’s definitely more than enough work in ensuring that you can be expert guide and a strong team mate. The product manager on my current team was brave enough to engage with the rest of the team like this and the results are really amazing. Instead of pushing product management on the team, he pulls the team into product management.

Also it should be obvious that when the team is spending time on things that are not software production that they will almost certainly release less lines-of-code. I’ve found instead though that the products developed this way are generally always delivering more value to the user, and doing so at a faster pace. No user is asking for “more software”.

Product Management Is Just One Possible Contributing Factor Though

In fact any of the disciplines on the team has the ability to cause the emergence of a waterfall model of development. It’s a systematic problem that is really tough to solve because it emerges from the socio-technical complex system of the software, the team, the company, and the market. As a result, there’s almost never just a single cause of it.

So with the obvious influence of product management aside, these additional properties seem to make waterfall vastly more likely:

Inability to see how traditionally downstream disciplines can actually occur simultaneously or even upstream.
Perfectionism (release-o-phobia) due to the perception that a particular stage is very expensive or irreversible.
An "Only specialists can do X" attitude (often due to perfectionism), sometimes worsened by an approval process requiring sign-off.
Lack of will to collaborate across disciplines. There's an inertia to collaboration that sometimes is hard to overcome.
Cargo cult adherence (eg "Such-and-such company does it this way", "This is the way I've always done it", "My software engineering textbook from 1968 says this is how you do it")

I've personally never seen waterfall-style development occur in the absence of these factors so I think about ways to stamp them out whenever I see them.

For the team that just doesn't know how to defeat waterfall but wants to, there are a tonne of things to try:

Create an environment of collaboration. Usually this involves collocated cross-discipline teams. When other companies are saying "Design is crucial. We should have a design team.", try to instead say "Design is crucial. We should have a designer on every team." It's this attitude that will have your team able to deliver software entirely on its own without endless meetings with other teams.
Make sure the team is surrounded by the information that it needs to be to make the right decisions. They should have easy access to your user-researchers, data scientists, customer-support, subject-matter experts, etc, etc to easily learn everything that they have to about their problem space. If you can put those specialists on the team as well, that’s even better.
Try to get the team to focus on a very small number of things at once, ideally 1. You want to focus on finishing instead of starting and getting each member of the team to do something different is the opposite of that.

Once you've achieved that proper environment, experiment with ways to disassemble the traditional waterfall order of things:

Have many people involved in all manner of planning (if they're interested... Some teammates may prefer to trust the others in their decisions and avoid all those meetings).
Have a common set of standard UI components so engineers can assemble pretty good designs quickly on their own.
Practice devops! It's the ultimate collaboration of developers, release managers and sysadmins often with all 3 roles rolled into one person (and heavily automated!) but at least on the same team.
Decouple release from deployment with feature flags and actually deploy before design is even complete. Often you can deploy all kinds of non-production-ready stuff constantly and only turn it on when it's ready. That's the essence of Continuous Delivery
Do testing BEFORE engineering. TDD and BDD are possibilities. If you have QA people, let them advise on what kinds of tests would be useful. That's what Quality Assurance actually means.
If possible, have everyone own testing. If possible, have engineers own test automation and test their own/each other's stuff.
Have lo-fi designs that are good enough to start engineering with. Let the hi-fi designs come later... maybe much later; Depending on your situation, you might be able to launch to 1% of your users with lo-fi designs and validate the product before investing in higher fidelity designs.
Have engineers and testers think through the lo-fi versions of the designs with the designer. Treat the designer as the team's expert guide rather than putting all the design responsibility on them.
Break up your work into smaller and smaller pieces. Even if you're doing some hand-offs, rework on small pieces is much cheaper.

The greater goal is to value results over effort, and to build the right software rather than the most software. Getting as many efforts in progress at once is an anti-pattern. Making sure everyone has something to do is a non-goal.

Without that perspective, many of these practices will seem inefficient. Sure sometimes you've got multiple people working slightly outside of their specialty, but the resulting software is generally always better, and all that expensive back-and-forth and rework is virtually eliminated. People are generally happier and more engaged as well because they know how important the work is and why.

_{Special thanks to Taylor Rogalski for feedback!}

So You Want to be a More Senior Engineer?

2017-06-04T17:22:43Z

"Engineer" really is a very silly term for people that make software. Engineering is supposed to be a predictable application of science toward predictable results. Software development is really anything but that. The biggest problem in our profession is and always has been getting predictable results.

The key difference between a junior engineer and those that are more senior is that the more senior engineer realizes the inherent unpredictability of delivering working software and constantly works toward making that delivery more predictable.

The junior, on the other hand, approaches the work with two key hinderances:

Being too intellectually challenged (or enamoured) with the details of programming in the small to be able to see the larger picture
Having little experience with complex systems (and often not even understanding that they're working in one)

As a result the more junior engineer will generally pursue solutions that risk predictability, frequently without realizing it. What's worse is that often these solutions will be faster/better/cheaper, further reinforcing the approach. Failures with risky approaches are often written off as fluke mistakes, technicalities, or the nebulous "human error" without reconsidering the methodology. Often they're written off like this even when the cost of the failures vastly outweighs the riskiness of the approach.

In software development, the details can seomtimes be really difficult. This never really goes away with experience, but the Junior Engineer hasn't yet been faced with that enough times to know it. And when the details are difficult, the junior engineer focuses primarily on the details, often to the detriment of the big picture -- predictability. It's a pretty understandable choice; our brains are limited in how much they can consider at once, and for some reason engineers have a tendency to default to concentrating on the technical details.

Conversely, the more senior engineer will be less encumbered by the technical details and will have been bitten a number of times by the nature of complex systems. If that engineer has taken the opportunity to learn from those mishaps, there's a chance for a much higher level of engineering (and predictability).

With that in mind, here are a bunch of examples that I think show the differences more plainly:

Junior: Finds a solution.
Senior: Finds the simplest solution.

Junior: Finds a solution for right now.
Senior: Considers the longer term implications when finding a solution.

Junior: Defaults to adding complexity/code with every requirements change.
Senior: Knows that changes often indicate a unifying concept and an opportunity to remove complexity/code.

Junior: Finds ways to manage complexity
Senior: Finds ways to remove complexity (because complexity management techniques are another form of complexity!)

Junior: Predicts results
Senior: Embraces unpredictability and manages it

Junior: Assumes that lack of evidence of problems is evidence of lack of problems.
Senior: Is aware that unknown unknowns almost always exist.

Junior: Can explain why their grand plan is flawless.
Senior: Ships as early and often as possible and judges solutions empirically.

Junior: Believes that it's sufficient to achieve quality at the edges.
Senior: Knows that often you have to ensure quality of internal components separately to achieve quality at the edges.

Junior: Believes that once the pieces are verified, the system is verified.
Senior: Knows there's still work to verify the system as a whole.

Junior: Will call "done" when the happy-path works, or the code is committed, or the code passes testing, or the code is in staging.
Senior: Realizes there are so many technicalities involved in actually solving real problems that one shouldn't call "done" until the user has used it and agrees.

Junior: Believes that once you get the software to work, it will work in perpetuity with no extra effort.
Senior: Realizes that maintenance always takes considerable time and effort.

Junior: Optimizes code that seems slow.
Senior: Measures for performance issues and optimizes only the bottleneck.

Junior: Writes code for the computer to understand.
Senior: Writes code for the computer and humans to understand.

Junior: Focuses entirely on the technical aspects of the role.
Senior: Realizes the entire complex system is a socio-technical one that also necessarily involves people. Constantly tries to also take the humans (managers, stakeholders, users, teammates, etc) into consideration.

Junior: Works tirelessly and continuously to solve a critical and time-sensitive problem.
Senior: Understands that communication during critical times is also crucial, and that "I have no update" is still a highly valued update.

Junior: Can explain why a solution is good.
Senior: Can explain why a solution is better than the others for a given criteria.

Junior: Is learning to be critical of software designs choices.
Senior: Is always trying to understand the tradeoffs in software design choices and can see the good in imperfect solutions.

Junior: Is learning new technology, design patterns, and practices and trying to apply them.
Senior: Judges technology solely on the basis of its ability to solve actual problems and skillfully avoids cargo-cult programming.

Anyway... these are just examples and certainly don't form an exhaustive list. I'm not at the end of learning from experience either, so I couldn't even write an exhaustive list if I wanted to.

One more example of the meta-variety:

Junior: Is starting to realize that predictability is valuable and is learning how to achieve it.
Senior: Realizes that different predictability approaches have different costs in different scenarios and makes choices accordingly after cost/benefit analysis.

The All-or-Nothing Principle

2017-04-09T16:07:00Z

^{Special thanks to my friends Deepa Joshi and Sam DeCesare for input/feedback!}

The 80/20 Rule, otherwise known as the Pareto Principle has wide-reaching implications for productivity. If you can get 80% of the results you want for 20% of the effort in a given endeavour it really lets you consider an efficiency/thoroughness trade-off that's right for your desired results. There's a real opportunity for saving a whole lot of effort and still raking in big results for that effort when you find a scenario that conforms to the 80/20 rule. Unfortunately there are a bunch of scenarios in life that don't conform to this nice 80/20 distribution and so I'd like to propose another one that I see pretty often.

The All-or-Nothing Principle: For many scenarios in life, there are 0% of the effects until there are 100% of the causes.

So while a Pareto distribution looks something like this:

...an All-or-Nothing distribution is much simpler; you've either got the effect or you don't.

It applies only to systems where results are binary which can also be found all over the place: everywhere from electronic examples like light switches or mouse-clicks, to the more biological -- like being pregnant or being dead.

Like the Pareto Principle, the All-or-nothing Principle can have a huge effect on productivity if you recognize it and handle it appropriately.

Here's a simple example: Bridges are expensive to build. Building a bridge even 99% of the way across a river gets you 0% of the results. This isn't at all the type of distribution that the Pareto Principle describes and so you can't make the same efficiency/thoroughness trade-off with it. You simply can't get 80% of the cars across with 20% of a bridge, so in order to extract maximum productivity from these scenarios, you have to approach the situation fundamentally differently. The important thing to remember in these scenarios (assuming you've identified one correctly) is that there is no difference at all in effect between 0% effort and 99% effort. If all you can give is 99% effort, you should instead give no effort at all.

Let's talk through the bridge example a little more: If you were a bridge-building company with 99% of the resources and time to build a bridge, you would be in a much better spot in general BEFORE you started at all than you'd be at the 99% mark, even though the 99% mark is so much closer to being complete. In fact, the almost-complete state is a sort of worst-case scenario where you've spent the most money/time possible and reaped the least amount of benefit. Where 80/20 distributions reap quite a lot of rewards for very little effort upfront, all-or-nothing scenarios reap no reward whatsoever until effort meets a specific criteria.

There are lots of examples of this in the real world too. Medical triage sometimes considers this trade-off especially when resources are constrained like in wars or other scenarios with massive casualties. Medical professionals will prioritize spending their time not necessarily on the people closest to death, but the people that are close to death that have a reasonable chance of being saved. Any particular person's survival is an all-or-nothing proposition. It's a brutal trade-off, but of course the medical professionals know that if they don't think they can save a person, they should instead spend their efforts on the other urgent cases that have a better chance; the goal afterall is to maximize lives saved. In the real world, probabilities about outcomes are never certain either so I'm sure this makes these trade-offs even more harrowing. (Interestingly, if a triage system like this allows 80% of the affected population to survive by prioritizing 20% of them, the effort on the population as a whole conforms to the Pareto Principle)

The All-or-Nothing Principle and Software Development

Fortunately for me, I work in software development where being inefficient (at least in the product areas I work in) doesn't cost anyone their life. All-or-nothing scenarios actually exist in dozens of important software development scenarios though.

Of all the scenarios, I don't think any has shaped the practices and processes of software development in the last 25 years as much as the all-or-nothing scenario of "shipping" or "delivering" product to actual users. The reason for that is that software only gets results when it's actually put in front of users. It really doesn't matter how much documenting, planning, development, or testing occurs if the software isn't put in front of users.

This specific all-or-nothing scenario in software is particularly vicious because of gnarly problems on both sides of the "distribution":

Expectations of the effort required to get to a shippable state can be way off. You never actually know until you actually ship.
Expectations of the effectiveness of the software to meet our goals (customer value, learning, etc) can be way off. Will users like it? Will it work?

Where more linear effort-result distributions (including even 80/20 distributions) get you results and learning along the way, and a smoother feedback cycle on your tactics, all-or-nothing scenarios offer no such comforts. It's not until the software crosses the threshold into the production environment that you truly know your efforts were not in vain. With cost and expected value being so very difficult to determine in software development the prospects become so much more difficult/risky. These are the reasons that a core principle of the agile manifesto is that "Working software is the primary measure of progress.".

It should be no surprise then that one of the best solutions that we've come up with is to try to get clarity on both the cost and expected value as early as possible by shipping as early as possible. Ultimately shipping as early as possible requires shipping as little as possible, and when done repeatedly this becomes Continuous Delivery.

Pushing to cross the release threshold often leads us to massive waste reduction:

Users get value delivered to them earlier, and the product development team gets learnings earlier that in turn lead to better more valuable solutions.

All of this of course depends on how inexpensive releasing is. In the above scenario, we're assuming it's free, and it can be virtually free in SaaS environments with comprehensive automated test suites. In other scenarios, the cost of release weighs more heavily. For example, mobile app users probably don't want to download updates more often than every 2-4 weeks. Even in those cases, you probably want a proxy for production release, like beta users that are comfortable with a faster release schedule.

We often find ways to release software to smaller segments of users (beta groups, feature toggles, etc) so we can both reduce some of the upfront effort required (maybe the software can't handle all the users under full production load, isn't internationalized for all locales or doesn't have full brand-compliant polish) but also get us seeing some value immediately so we can extrapolate as to what kind of value to ultimately expect. The criteria of even releasing to one user has a surprising ability to force the unveiling of most of the unknown unknowns that must be confronted for releasing to all users, and so it's a super-valuable way of converging on discovering the actual cost for delivering to all users. I've personally found these practices to be absolutely critical to setting the proper expectations about cost/value of a software project and delivering reliably.

Furthermore, there is a lot more to learning early than just the knowledge gained. Learning early means you have the opportunity to actually change what you expected to deliver based on things you learned. It means that the final result of all this effort is actually closer to the ideal solution than the one that loaded more features into the release without learning along the way.

Navigating All-or-Nothing Scenarios

Again, the benefit of identifying All-or-Nothing scenarios is that there are particular ways of dealing with them that are often more effective than solutions that would be common with other types of scenarios (like 80/20 scenarios). I'll try to dig into a few in more detail now.

Try to make them less all-or-nothing

We've seen from above how continuous delivery breaks up one all-or-nothing scenario into many, thereby reducing the risk and improving the flow of value.

More theoretically, it's best to try to change unforgiving all-or-nothing scenarios into more linear scenarios where possible. That completely theoretical bridge builder could consider building single lane bridges, or cheaper, less heavy-duty bridges that restrict against heavier traffic, or even entirely different river crossing mechanisms. These are just examples for the sake of example though... it's probably pretty obvious by now that I know nothing about building bridges!

Software is much more malleable and forgiving than steel and concrete, so us software developers get a much better opportunity to start with much lighter (but shippable) initial passes and to iterate into the better solution later. Continuous delivery, the avoidance of Big-Bang releases, is an example of this.

Minimize everything except quality

In all-or-nothing scenarios, reducing the scope of the effort is the easiest way to minimize the risk and the ever-increasing waste. It can be tempting sometimes to consider also reducing quality efforts as part of reducing the scope, but I've personally found that reducing the quality unilaterally often results in not actually releasing the software as intended. The result is that you leave a lot of easy value on the table, don't capture the proper learnings early, and set yourself up to be interrupted by your own defects later on as they ultimately need to be addressed. If you don’t really care about the quality of some detail of a feature, it’s almost always better to just cut that detail out.

Realize that time magnifies waste

For the types of work that need to be complete before the core value of an endeavour can be realized, there's not only a real cost to how long that endeavour remains incomplete, but the more it becomes complete without actually being completed, the more it costs. In classical Lean parlance, this is called "inventory". Like a retail shoe store holding a large unsold shoe inventory in a back storeroom (it turns out that selling a shoe is generally an all-or-nothing scenario too!), effort in an all-or-nothing scenario has costs-of-delay. A shoe store's costs-of-delay include paying rent for a back store-room and risking buying shoes it can't sell. It's for this reason that Toyota employs a method called "Just-in-time" production. It's the cornerstone of Lean manufacturing (that has coincidentally also had a huge influence on software development!).

In the case of software, larger releases mean more effort is unrealized for longer. This results in less value for the user over time and less learning for the company. If you can imagine a company that only releases once a week, high-value features (or bug fixes) that are completed on Monday systematically wait 4 days before users can get value from them. Any feature worth building has a real cost of delay.

Similarly, more parallel work in progress at once means there's multiple building inventories of unfinished work. If a team can instead collaborate to reduce the calendar time of a single most-valuable piece of work, this waste is greatly reduced.

Stay on the critical path

In all-or-nothing scenarios, it's crucial to be well aware of the critical path, the sequence of things necessary to get you to the "all" side of an all-or-nothing scenario. These are the things that will affect calendar time and affect how quickly you learn from users and deliver value. In software engineering, this is often called the steel thread solution. If you find yourself prioritizing non-critical path work over critical path work (which is common if you're not ever-vigilant), you'd do well to stop and get yourself back on track.

Pick your battles!

If you're faced with an all-or-nothing scenario, and you know or come to realize that you can't make the 100% effort required, the most productive thing you can do is to not even try. Like the 99% complete bridge, your efforts will just be pure waste.

Sometimes this can be extremely hard. It's why the term death march exists for software projects and project management in general. People just have a really hard time admitting when a project cannot succeed, even when they know so intellectually. It can be particularly difficult when the project has already had effort spent on it, due to sunk cost fallacy, but that doesn't change the proper course of action.

If you find yourself in a truly unwinnable all-or-nothing scenario, it's best to forgive yourself for not pursuing that project anymore and to put your efforts/resources somewhere with better prospects. It's not quitting -- it's focusing.

So You Want to be a Manager?

2017-03-06T19:56:05Z

^{Special thanks to my colleagues Sam Chaudhary, Lexi Ross and Stuart
Reavley for feedback/criticism!}

I’m not a management guru, and I’ve never taken a management class. I have done some management, and a hell of a lot more coaching (which is often more powerful than traditional management) though, and I’ve worked with and for managers of varying degrees of effectiveness and I’ve picked up a few things that I’ve seen work really well.

While I’ve got a lot of experience with management of technical systems, I’m really just going to talk about the management of people here as if it’s the only type of management. It certainly isn’t, but it’s important enough to attempt to discuss separately.

Management is sometimes a title, but it’s more important that you think of it as a responsibility. It’s the responsibility to be effective in the larger organization.

Let’s break that down further: being effective means achieving the desired results for a given endeavour. When you’re not a manager, you can often get away with just trying hard and waiting for someone else to correct your course when the results aren’t sufficient (though I hope it’s obvious later that I don’t suggest this). Conversely, managing requires watching results tirelessly and changing tactics that aren’t working until they work.

Congratulations, You’re Hired!

You’re already in a management position: No matter who you are, you’re in charge of managing yourself, and the world is constantly deciding how it values you based on your ability to be an effective person. Even if you were to become a hermit living on a remote island somewhere, you have to constantly be considering your effectiveness at how you spend your day in order to survive on a daily basis. When you see your shoelace untied and you tie it back up, that’s self-management. Before you can manage endeavours that involve other people, you’ll want to make sure you’re reasonably good at managing yourself.

Managing Without Being a Manager

A team can successfully be self-managing in varying degrees, or a team can be managed by a single individual. My preference has always been to lean as hard on the team as possible to be self-managing but regardless, “management” as a responsibility (ensuring effectiveness) must happen.

It is also possible to manage sideways and to manage up. This means that you are ensuring that management is happening regardless of your title. At any company I’ve seen, people that do this successfully (they’re effective and not an asshole) are the natural choices for management roles, and they get them much more often. You have to constantly pick high-value problems, and work within your organization to solve them.

The easiest way to start trying to manage up is to be part of your own solutions. When you find a problem that you can’t solve yourself and want to bring it to your manager, bring some potential solutions too.

Part of managing is finding your own high-value problems to solve. More commonly this is called “taking initiative”. If you’re waiting for someone else to give you the opportunity to “take initiative”, you’ve got a fundamental misunderstanding of what “taking initiative” means.

People that “want to be a manager” without having previously managed sideways or up are continuously overlooked for that role. If you can’t be successful at this without a title, you’re probably not going to be successful at it with a title either. Management isn’t for everyone; it often means a bunch of administration, bureaucracy, and conflict.

7 Core Principles

There are 7 principles that I think are important to pursue constantly and relentlessly to achieve effective management. Theoretically none of these are actually necessary for delivering results, but I’ve never seen a team consistently deliver results without them.

Integrity

Help the team identify and adhere to team values and goals.

Do what you say and say what you do. Relentlessly change what you’re doing to match what you’re saying you’re doing, or change what you’re saying you’re doing to match what you’re doing.
Be trustworthy and internally consistent.
Ensure that the entire team is acting as a cohesive unit.

Reliability

Reliability means rejecting responsibilities that you can't be successful at.

When you find you’ve got a responsibility that you can’t fulfill, raise the issue externally immediately.
If you find this is happening repeatedly, figure out why and stop it.
When you find that the organization is exerting pressures on the team that are counter-productive, defend the team by fighting to create space for it to be productive and happy.

Alignment

Ensure the company and product vision, mission, and the strategy to achieve them are unambiguous, well known and widespread.

Work tirelessly to ensure buy-in from team-members, whether that involves changing minds or changing goals.

Two important tips for easier buy-in:

Make sure people know WHY the goals are what they are. Even if they’re happy to work on any goals, the reasoning behind them can make them much more effective.
Getting people’s input on goals is the easiest way to get buy-in, and a great way to get better goals.

Transparency

Ensure progress and status on the goals are well known and widespread.

ALWAYS keep interested parties up-to-date. Saying “I have no update” is almost always a crucial update.
Don't be surprising. ALWAYS ask yourself "Would the current state of our progress be a surprise to interested parties?".
Favour processes, practices, systems where transparency is built in, so it is less tedious (Nobody wants to do manual status reports, and that dislike is often an impediment to achieving transparency).
Make failures or difficulties transparent to the larger organization, including your efforts to solve them.
Effective communication is the key. Ensuring that the audience has actually understood it as easily as possible is critical. Do not waste your time writing documents/memos that people don’t read or don’t understand.

Continuous Improvement

Support continuous improvement in processes and practices.

Regularly revisit the current practices and processes and look for ways to improve.
Throw out practices and processes that cost more than you get from them.
Revisit practices and processes after large projects or new problems.
Consider ways to double down on successes and reduce failures.
Consider not only prevention of failures, but also mitigation of failures when they occur anyway.
Find ways to measure success and failure and make it more constantly obvious to everyone so they can adjust faster.
In general “try harder next time” is a recipe for repeated failure. Think of ways to try differently.

Empowerment

Ensure that the team is set up for success and motivated.

Often this includes absolutely crucial human issues like staffing, team composition, interpersonal conflicts, and individual compensation.
Happy, well-supported teams are effective teams. Continuously poll the team to find out how to better support them.
You’ll need social and emotional intelligence to tackle this, and there’s no shortcut around it.
People are set up for failure when their responsibility in a given situation exceeds their ability and control in that situation. Root out these situations and rectify them immediately.
Be gracious, thankful, and celebratory where warranted. Genuine appreciation is a powerful motivator.

Autonomy

Avoid being a bottleneck by delegating all possible responsibilities back to the teams.

It’s exceedingly difficult for one person to beat the ideas and efforts of an entire team, so it’s foolish to not promote as much shared-responsibility and autonomy as possible for the capabilities of the team. Coaching (as opposed to command-and-control management) is the extreme logical conclusion of this.
Fix things within your realm of capability. Raise all other issues (ideally with possible solutions) where possible.

And with all that said...

Usually when I’ve been ineffective (which occurs to some degree on a daily basis), I’ve found that it’s because I’ve failed to achieve one of the above principles. I still struggle with all of these quite regularly myself, and I’m not sure there’s ever a point where they’re “mastered”.

Most of these principles can and should be applied to sideways or upward management as well. Also, see how many you can apply to yourself; at the very least you’ll be a better teammate.

Ultimately there are no points awarded for effort. Effectiveness requires actual results, so even if these principles aren’t getting success for you, it’s your responsibility to throw them out and come up with better ones.

The Quality Feedback Loop

2017-01-15T05:22:07Z

A lot of engineering management and product development process conversation tends to be based on a fundamental assumption that product quality and development speed are always opposing forces. Time and again however I find myself learning and relearning that speed and quality can instead be symbiotic and improving one can also improve the other. These win-win scenarios are actually everywhere as long as you've got your mind open to the fact that they're possible.

One place where I think that this is obvious is the multiple feedback loops around quality. New features go through these loops over and over and so it's hugely important to optimize them if you want to ship software quickly.

There is also a sometimes argued axiom in software development that bugs found sooner are often the cheapest / fastest to fix. I do agree that the empirical data on the subject is a little light, but I think it follows naturally from the fact that longer feedback loops make systems harder to manage and reason about. For example, if your only quality measure for anything you do is customer reports and you do nothing else for quality whatsoever, solving issues is extremely time-consuming and error-prone.

So lets start with that outer loop and list a bunch of other common quality feedback loops:

Customer reported bug
Logged error seen during regular inspection
Error alert happening when a defect occurs
Defect Found by testers before launch
Broken CI build
Broken test suite
In-Editor error

Customer reported bugs

These are of course the most expensive; The feedback loop is largest here. If you're maintaining a large feature set, it's possible that you don't remember the details of how a feature works, or even that the software has that feature at all. These defects also come back at inopportune times and interrupt you while you're working on other tasks. You're likely also on a team so the defect probably isn't even one that you had any hand in. Lacking all of this context makes solving the defect all the more difficult and all the more time-consuming.

There's the obvious cost to the user as well. Production defects can cost you customers, or in extreme cases can even lead to lawsuits.

Ultimately you want to find a way to shorten the feedback loop, which usually means trying to move this to an early feedback loop.

As long as the measure that tightens the feedback loop is cheaper effort-wise than the defect, you've got improved speed and quality. It almost always is cheaper though because these loops are run many times for each feature (though there are some common types of loops that are especially expensive, like manual human regression testing).

Logged errors and metrics seen by regular log inspection

Production logs are a critical part of making an application self-reporting. If you're regularly checking your logs (and keeping the signal to noise ratio in them high) there's a good chance of finding production defects before users do.

That's great because it can catch things weeks or months before customer reports do sometimes, and that faster feedback loop means you're more likely to remember the affected area of code.

Usually for these types of issues though, we can go one level deeper...

Error alert happening when a defect occurs

If you have a system set up where logged errors increment a metric, you can find a way to put an alert on a threshold for that metric. There are a bunch of services you can integrate for this functionality, or you can run your own kibana service. The point is that your production systems can be self-reporting: they can tell you when there are problems, thus tightening the feedback loop further.

This is also super useful, because it really helps your mean-time-to-repair. Minimizing the amount of time it takes to find a defect in production also helps minimize the amount of time a user is affected by that defect.

Defect Found by testers before launch

Unfortunately I think the most common way of finding defects is manual human inspection. It's a natural choice of course, but it's by far the slowest and most error-prone. It's a valid method if you can't solve your issues otherwise, but the repeated compounded cost including the time to test, and the way it affects your ability to quickly deliver software shouldn't be ignored. When a good automated test is possible, it'll be both faster and less error-prone. I work on a production system that has ~4500 automated tests that run dozens of times per day. Having humans do that is impossible.

With all that said, these are still defects that are found earlier than in production and so they save your customers from the defect, and they lead to a tighter feedback loop. It's just that this feedback loop is so expensive that as a developer you really shouldn't be relying on it the way you can rely on even tighter feedback loops.

Broken CI build

The first line of defence after your work leaves your machine is the CI build. Any quality measures you have in your build process (which I'll get into shortly) should be part of this build and they'll verify that what you've got in the main branch is ready to move on to. If the main branch doesn't pass the same barrage of quality measures as the local machine build is supposed to, it certainly shouldn't move past this step on its way to production. It could be that this is your last line of defence before affecting customers, or it could be that you have a human tester that can at least know not to bother testing a broken build (Ideally, passing your quality measures is necessary for any build-artifact to exist at all so that testing a broken a build isn't even an option).

Of course this is an easy savings to your users and any human testers you might have, but it's still not the tightest feedback loop you can have. It's also an expensive measure to your team-mates; it means there's a period of time where the main branch is unusable, blocking their work.

CI builds really should be doing the same thing as a local developer machine's build so that developers have a reasonable assurance that if they run the build locally and it works, it should pass in CI as well. Let's talk about some of the quality measures that should go into a build.

Broken test suite

Automated tests can make up a feedback loop that is almost instant. On most platforms that I've worked, testing is fast enough that a single test almost always takes less than a second. I work on a codebase with around ~4500 automated tests that runs in about 2 minutes (albeit due to herculean efforts at parallelization). The speed of these is super important because it makes the feedback loop short and helps prevent developers from relying on CI as a personal build machine.

Comprehensive test suites are expensive! We spent a lot of time maintaining ours, adding to it, and ensuring it stays fast. It's almost certainly our most effective quality measure too though.

Integration tests tend to be faster to write, because they test more things at once, but when they do fail, you've usually got some extensive debugging to do. Unit tests tend to take more time to write if you want the same level of coverage, but when they fail, you usually know exactly where the issue is. These are things to factor into your feedback loop considerations.

There are still tighter feedback loops that are cheaper to maintain though and those should be relied on where possible.

In-editor error

Any type of static analysis that can be performed in your editor/ide, like linting or static type-checking is an even tighter feedback loop still. Any problems are evident instantly while you're still in the code and it indicates to you exactly where. This is an extremely fast feedback loop that you'll probably want to employ where it's possible and it makes sense.

I don't have a tighter feedback loop than this, but in some cases you can still do better...

Abstraction that simply makes the error impossible

If it's possible to use tools/abstractions that make the defect impossible, that beats all the feedback loops.

Some examples:

Avoid off-by-one errors in for loops by using iterator functions, or functional paradigms that give you filter(), map() and reduce().
Avoid SQL injection by using prepared statements.
Avoid Cross-site scripting attacks by using an html template language that automatically escapes output.
Avoid bugs from unexpected changes in shared objects by using immutable data structures.

Working through the levels

These levels all form a sort of "onion" of quality feedback loops where the closer you get to the middle, the cheaper the defect is.

Thinking this way, you can easily see how if your users are reporting an issue caused by a sql injection attack, you would ideally work to push that problem to tighter and tighter feedback loops where possble. If you can make it show up in logs or alerts, you can fix it before users report it. If you can have testers test for it, you can fix it before users are subjected to it. If you can write some unit tests for it, you can save your testers from having to bother. If you can use the right level of abstraction (prepared statements / parameterized queries in this case), you can eliminate the class of error entirely.

Delivering high-quality software quickly means looking at the most expensive, time-consuming or frequent classes of errors and systematically pushing them to a lower-rung in this onion of quality feedback loops. With a little situational awareness and a little creativity it's almost always possible and leads to huge cost and time-savings over the long haul.

This is just one of the many ways that I think the speed vs quality dichotomy in software engineering is a false one.

The Atomic Team Pattern

2016-11-16T18:11:16Z

One pattern that I've seen work well in software development management is one I call The Atomic Team pattern. Basically what it means is that ideally the smallest indivisible unit for management is the team. I think it's a cornerstone of healthy service-oriented architecture, but it's rarely talked about because it's on the more touchy-feely and less-technical side.

Management of such a team ends up primarily involving:

Communicating expectations and objectives
Ensuring the team has what it needs to be successful
Handling (ideally) infrequent interpersonal issues that the team can't handle themselves.

Disadvantages

There are a few disadvantages:

Teams can take some time to form.
Some developers don't have the interpersonal skills for this.
There are always interpersonal issues to consider.
Teams are not resilient to reorganization.
Will individuals on the team be happy with their role on the team and the type of work that the team takes on?

Advantages

Requires less management

Let the team self-organize around tasks instead of being command-and-control orchestrated. Drop prioritized work into the team work queue and they can figure out who does what and when based on their own availabilities and skillsets. Team members can hold each other accountable for the team objectives.

Lowered Bus-Factor

Cross-functional teams often have more than one person capable of doing a given task. This keeps the queue moving even when someone is sick or on vacation.

Less Stress for Team Members

On a well-formed team, team members can count on one another for help and support. Vacations and sick days rarely affect a delivery timeline much at all. No one needs to be the only one responsible for stressful issues, outages, deadlines, or un-fun work. The increased autonomy in how the team executes on its objectives is highly motivating.

Better cross-pollination of knowledge and skills

Team members can teach each other skills and organizational knowledge so that they level each other up. My best career development has come from working with brilliant peers.

Better solutions

Put simply, two heads are better than one. The best solutions that I've seen in software development are most often devised by a team riffing off each others' ideas with a whiteboard.

Long-term group ownership of codebases and services

In reality, software doesn't just get deployed and run in perpetuity without engineering intervention. Even if there aren't any new features to add, there are hardware issues, bugs, security issues, and software upgrades. When teams own the software they write forever, they write better software, they monitor it better, and they manage it more easily. Without team-ownership, you're often left looking for someone to work on some long forgotten service when it inevitably needs work.

The result of these advantages is an software development organization that scales much more easily.

Anti-patterns when Managing Teams

Team-Busting

Examples:

adding / removing the members of the team often
loaning out team-members for other non-team objectives
holding individuals responsible for results of team objectives
creating competition within the team

These types of activities remove the mutual trust and comradery from the team and stop the team from forming beyond "just a group of people". This stops the members from working as closely together as they could otherwise and undermines a lot of the advantages of having a team.

Over-Management / Micro-Management

Examples:

dictating who on the team does what
becoming a communication conduit for two team members when they can just talk to each other

These types of activities are mostly unnecessary except in extreme situations so they're a time-sink and create a bottleneck that makes scaling your organization harder. Additionally, a manager can often make much worse decisions in these areas than the team members because they're the people closest to the team and the work.

Not Respecting Communication Overhead

Examples:

Choosing teams of similarly skilled individuals (eg "The Frontend Team"). Teams that are not cross-functional often can't deliver their own solutions without a lot of communication with other teams.
Choosing teams that are too large. Teams that are too large spend too much time coordinating efforts and coming to consensus on issues.

We're in the Stone Age of Software Engineering Management

2015-06-15T03:20:37Z

Obviously from the title, I think software engineering management is in a Stone Age. Before I get into my arguments though, I'd like to say that this isn't really about any particular manager or managers that I've had in the past. It's really about counter-productive patterns that I've seen that I really think we need to evolve from. I've also been responsible for some of these mistakes myself as a manager or technical lead over my years in software engineering, so if this comes off preachy, that's not the intention.

There are a few common problems in engineering management that keep us in the Stone Age though, and I'd like to detail some of them.

The longer you manage, the less technically competent you become.

"Technology is dominated by two types of people: those who understand what they do not manage, and those who manage what they do not understand." -- Putt's Law

It's undeniable that knowledge work has some fundamental differences from factory work: there's never a point where you've completely learned how to do your job, and in fact the knowledge required around your job is constantly changing. For this reason, the longer you're not developing software, the worse you get at it. Being able to type code is only tangentially necessary for software development. The real job is making decisions about how the software should work.

This is one of the few cases in industrialized production where the worker quite regularly knows more about their job than their manager. And if the manager does happen to know more, the gap between them will be constantly narrowing.

The impact of this fact is pretty wide-reaching. It means that a manager is generally not going to be useful for their technical competence, or at least that their usefulness will be waning. In my career I have rarely met exceptions to this rule.

Management is viewed as a promotion and not a distinct role with a separate set of skills.

I'd like to propose instead that management is a distinct role that requires a different set of skills, none of which require the ability to make technical decisions (and I'll get to those later).

Unfortunately management in the software development industry is looked at much like management in other industrialized production. Management is widely considered to be "a promotion" that developers should aspire to. It's a common expectation that developers at a certain level of seniority (or age) should be moving into management. There's a very real pressure from the rest of the world too because they view management as a "higher" position, so you often see developers pushing for this "promotion", even though they have not acquired the disparate skill-set required of a manager. The result is that often companies trade excellent developers for terrible managers, and often to the detriment of those developers as well.

Management is viewed as an imperative to command and control.

It's common to move people into management based on technical ability, because surely we must need a technical person to make the important technical decisions, right? That's certainly the argument that's most often made to support the management-as-promotion mindset.

There are huge drawbacks to the manager being "The Decider" in technical matters though (notwithstanding their continually eroding technical ability):

Even extremely technical managers are less likely to have better ideas than their entire team.
This is not a scalable solution. Even on a small team, the manager will not have time to make all the decisions because aside from typing, all software development is fundamentally about making decisions. As the team grows, the manager will be capable of making an ever decreasing number of decisions. Some managers try to mitigate this by carving out which decisions they will make (the important ones!), but this makes them a process choke-point, and increasingly a productivity net-negative.
With decision-making being the developer's primary job, a manager that tries to make decisions is taking away the most interesting part of the job for the developer, and with this their feeling of autonomy, responsibility, accountability and ownership. The result is that this type of manager is actively and powerfully demotivating their team. Ironically, command and control managers seem to be the ones most irritated when their team doesn't self-organize in their absence. In reality it's quite a long and difficult cultural change for a "controlled" team to become a self-organizing team and I've only seen the presence of a command-and-control manager to be an insurmountable impediment.

It certainly doesn't have to be this way, but many view management not just as a chance to command and control, but also as a responsibility to command and control. If the team is allowed to make technical decisions together though, on their own, all of these particular problems can just disappear. And I would argue that if a manager doesn't have a team that can be trusted with this responsibility, it is the manager's responsibility to transform that team.

Interestingly, it's a pretty normal refrain from engineers that they want their manager to be technical. I believe that much of that sentiment comes from an assumption that management is command-and-control by definition, and of course the worst of both worlds would be a non-technical command-and-control manager (and I whole-heartedly agree). I also think that if management would instead trust technical decisions to those most close to the work, engineers would be far less likely to care about their manager's technical competence.

Management focuses on enforcing the predictability of software development rather than on mitigating its undeniable unpredictability.

Software development is development of complex systems that are, by definition, extremely difficult to predict. Typically management tries to predict, control, and reprimand. Given the fact that complex systems are resistant to those types of actions, the effort is largely fruitless and often counter-productive.

Some examples:

Deadlines are one such mechanism of control. Despite the fact that engineers are the best people from which to get accurate estimates, it is extremely common for managers to dictate deadlines for fixed-sized projects, and to repeatedly find their team failing to meet them. For some reason, this repeated failure is not an indication that something has gone wrong management-wise. Often the team is chastized, over-worked, or reprimanded instead. Of course the real world will have dates after which delivery of software greatly diminishes its value, but deadlines are simply a thoughtless and reckless way to enforce either minimizing a feature or lowering the level of quality that goes into it (usually the latter). When a business uses deadlines with a fixed scope of work and quality, it's willfully ignoring the reality of its team's capability.
Estimation is an example of attempting prediction. Complex systems are inherently difficult to predict as well, and we've all seen evidence of this in our own inaccurate estimates. The best success we've had with improving estimation is to limit the amount of work we bundle up in an estimate which is to say we've improved estimation by estimating less. The sum of a project's timeframe often ends up being greater than the sum of the timeframes of its parts though, because we often even fail to predict all the pieces of work that are necessary.

Estimates from developers, when treated as self-imposed deadlines, are really no better than deadlines, because prediction of how long a software project will take is so incredibly difficult and developers are rarely given the training or time to do better estimates. Estimates like this end up being another mechanism of control, but a somewhat more insidious one, because developers will feel like they have no one to blame but themselves for "missed estimates".

Reprimanding developers for defects or service outages is another often counter-productive mechanism of control. As with many complex systems, there is never a single root cause, and the humans and the technology are so inseparably intertwined in the system that it means nothing to conclude "human failure" and reprimand the human. That's just not a results-oriented path to improved quality. I'd never suggest that post-mortems are a waste of time, but they are when conducted in a fashion that stops the investigation at "human failure" or even after a single cause. Investigations like this result in the workers feeling terrible, and the process as a whole not actually improving.

All of these management efforts are time-consuming, stressful, and mostly counter-productive (in that they take away from the developer's time to write software).

Of course you might reasonably ask how the feasibility of a proposed feature can be assessed without some estimation of its costs, which is a fair question, but an effective manager must realize that the estimate (as well as the scope of the work being estimated) must be continually revisited and revised and the stake holders need to have their expectations continually managed/adjusted.

The complex system that the developers create includes all the people involved.

Anytime something happens in the system, it can't realistically be viewed solely from a technical perspective. The technical aspects exist in large part because of what developers did or did not do to make them happen. The interdependencies between people and the technology forms in such a way that most mental models that separate them are not grounded in reality. There are a few results of this:

Developers can't simply be swapped in and out of the team without significant costs. They form relationships and interdependencies with both the technology and the other people on the team that are time-consuming and difficult to reform. This cost can be minimized by having the team regularly push to make itself pan-functional (usually via intra-team mentoring).
Conversely, a developer can have real negative impact on this socio-technological system in many ways, the most detrimental of which is by not acting in a way that's worthy of the team's trust. If a team member regularly refuses (explicitly or implicitly) to act in accordance with the team's general wishes, that developer must be removed, regardless of some abstract notion of "technical skill level" or how "indispensable" they've made themselves.
The team itself requires maintenance. They need to be encouraged and allowed to regularly take the time to look at their process and practices and decide what steps they should take to improve them. They need to be supported in their efforts to take those steps.
A high performing engineering team is more valuable than the sum of its highly performing individuals. A team needs to be cultivated carefully with an eye on creating the desired culture of collaboration and trust. This is extremely difficult and takes a great deal of time and effort, but it pays off wildly with much smarter decisions and much faster development.
Software development cannot be broken into an assembly line with specification, coding, and testing stages. All of these things occur and reoccur as necessary. They are all "development" and there's nothing to gain by pretending that the task is divisible.

The complex system that the developers create has a lifetime.

As a piece of software is used more and more over time, the chances of a user finding a way to put the software in an unexpected state increases. It's even common for new bugs to arise in narrowly scoped software that's over 20 years old.

Ultimately the only software that doesn't require maintenance is software that no one uses anymore. Otherwise, there's no such thing as completion of a piece of software. At best you can get a convergence toward completion, assuming the project is narrowly scoped and you can resist feature-creep.

Unfortunately management typically looks at software development as a series of projects or features with definite endings where there is an expectation that no more work will be necessary at the completion of the "project", and that the team will be 100% available for the next project.

This fallacy may or may not lead to completely unrealistic expectations on the first project or two, but as the number of projects "completed" increases over time, the team will become busier and busier maintaining those projects and the team will necessarily get slower and slower.

There are innumerable counter-measures that can be taken to ease the maintenance of software over time of course, but most of these require management to first realize that this reality exists, and to allow the team to spend the time to take these counter-measures.

Management that doesn't realize this often misses the point of spending time on anything that doesn't yield value immediately. For example, automated tests are in many scenarios a super valuable way of ensuring regressions don't creep in. They often don't provide enough value to offset their immediate up-front costs, but I've rarely seen them not be a net positive over time as they eliminate the need for slower manual testing forever after. Short-sighted management will be reluctant to make investments like this, and therefore doom the team to lower productivity over the long haul.

Goals and priorities are rarely clearly and intelligently set.

Goal and priority setting is the absolute number one deliverable that management has for the team. Unfortunately it is common for management to busy itself with other less productive tasks, often involving micro-management, and to actively disrupt the team's effort to ensure that goals and priorities are clear.

Some common ways that management fails at this:

interrupting the team working on the top priority goal to talk about or pursue a lower priority goal
not ensuring that the team is shielded from lower priority goals from other parts of the organization.
failing to ensure that work is constantly presented in priority order
failing to ensure that there is some semblance of reasoning behind the order of priority
quickly oscillating between priorities frequently keeping a team starting new things, and rarely finishing anything.
failing to coordinate priorities between teams that have dependencies on one another.

The most common failure I've seen here is management's unwillingness to choose a more important goal between two top goals. It's fine for management to ask for help from the team in that prioritization, or even for management to pick one randomly (if they're really so equal, it shouldn't matter), but what is unacceptable is for management to say "These are our two #1 goals". In that case, the team is forced to take over this management responsibility, because any one person can only do one thing at a time. If the team has not learned to manage that responsibility themselves, they will often be terrible at it. Some members will be working on one thing and others will be working on the other when just a touch of management could have had them coordinating to get one thing done first and delivering value as soon as possible. Instead the manager in these scenarios has ensured that value will be delivered in a slower way than what's optimal.

Efforts and Practices are rarely critically examined with any attention to the results.

It's unfortunately extremely uncommon for managers today to pay much attention to the actual results of the team's effort and the practices that it follows.

I think that a lot of the reason behind this is that it's extremely difficult to admit that things aren't going well when we've tied those results to our self-image. The only thing worse than mistakes though are mistakes that go uncorrected. Unfortunately the first step to correcting a mistake is to admit it exists.

Ego can also get in the way and take management off into more interesting (or brag-worthy) endeavours than what's best for the business. I've seen this happen countless times where teams deliver absolutely nothing while working on the latest in tech. If the same ego-driven management style reaches high enough into the organization, and the company is profitable enough to support it, it's easy for a team to get away with this blunder for years (which I've also seen).

Copy-cat or Cargo-cult management is probably the next most common excuse for not examining results. Often I've heard "google does x" or "y is an industry best practice" or even "that's not Agile" without any discussion of whether a particular practice makes sense for the team's own particular goals, scenario, or needs. Often these managers feel that they're adhering to "best practices" and so the actual results will necessarily be optimal and won't need to be examined. I'm definitely a proponent of agile methodologies in general, but saying whether or not something is "Agile" explains nothing about it's utility to the organization. There are no sacred practices that should be allowed to escape scrutiny. Any practice that can't be shown to be adding value should be canceled immediately.

It's common as well when a manager does manage to pay attention to results that his/her methods are extremely flawed. Probably the most infamous example of this is how managers used to count lines of code to judge the productivity of a developer. Counting the number of commits is also ridiculously flawed in the same way: they're missing what it means to be productive in an engineering context entirely. The least productive engineers are doing things like building needlessly elaborate architectures and suffering from NIH when they could just use a far better 3rd party library. Yes, you do need to be able to determine if a developer is productive or not, but that doesn't mean that that determination can be quantified. There will be some very important things that you need to measure that aren't quantifiable, so managers need to be comfortable with qualitative measurements.

In short, I think it's pretty common for engineering management to be actively harmful to their team's speed and their product's quality. Because uncorrected management mistakes impact entire teams, it's quite easy for a manager to have an overall net negative contribution to the organization. Of course it doesn't need to be like this, but a lot of the common expectations of management will really need to change first.

Reclaiming Value from Bugs and Outages: Thoughts on Post-Mortems

2015-03-08T16:12:23Z

_{(Thanks to Chris Frank and Sam Decesare for the feedback!)}

Bugs and outages happen. If the team can learn from them though, it's possible to reclaim some of that lost effectiveness, sometimes even to the point that the learning is more valuable than the effectiveness lost.

The best mechanism I've seen to drive that learning is for the team to meet after the outage or defect is no longer an immediate problem and for the team to try to learn from what happened. This meeting is often called the Blameless Post-mortem. These meetings have been the biggest agents of improvement that I've ever seen on an engineering team, so if you think you'll save time by skipping them, you're probably make a huge mistake.

Institute Blamelessness

It's crucial to assume that everyone involved in the issue would've avoided it if possible, given the chance again. It's natural for people to want to point the finger (sometimes even at themselves) but if the team allows this, it'll quickly lead to a culture of cover-ups where nothing can be learned because real information about what happened can't be extracted.

What's worse than this is that when you decide who's responsible, you have no reason to continue to investigate what there is to learn. It seems obvious that since that person is to blame, if they're either removed or coerced to try harder next time, the issue won't reoccur.

This reminds me of reading about airplane crashes in the news when the airline concludes the cause was "human error". That was really the best that that airline could do? They decided to either ask that human to be better next time, or replaced that human with another human. There's really nothing to learn? No way to improve the aircraft? No way to improve the process? No way to improve the environment? No way to improve the human's understanding of these? This is an airline I'd be afraid to fly with.

Sidney Dekker's The Field Guide to Understanding Human Error, an absolutely genius book on the subject, sums it up nicely: "Human error is not the conclusion of an investigation. It is the starting point."

Look for all the Causes

Often these meetings are called RCAs or Root-Cause Analysis meetings, but I try not to call them that anymore, because there never really ever seems to be a single root cause. Computer systems are complex systems, and so usually multiple things have gone wrong in order for a defect to appear. John Allspaw explains it more succinctly:

Generally speaking, linear chain-of-events approaches are akin to viewing the past as a line-up of dominoes, and reality with complex systems simply don’t work like that. Looking at an accident this way ignores surrounding circumstances in favor of a cherry-picked list of events, it validates hindsight and outcome bias, and focuses too much on components and not enough on the interconnectedness of components.

Also make sure you've validated your assumptions on these causes. There's a powerful tendency to fall prey to What-you-look-for-is-what-you-find principle for the sake of expediency and simplicity.

Dig deeper

It's usually a good idea to dig deeper than you'd expect when determining how a problem occurred. One practice that forces a deeper investigation is The 5 Whys. Basically you just ask "why?" for each successive answer until you've gone at least 5 levels deep, attempting to find a precursor for any problem and its precursors. Often in the deeper parts of this conversation, you end up investigating bigger picture problems, like the company's values, external pressures, and long-standing misconceptions. These deeper problems often require internal or external social conflicts to be resolved, which often makes them tough, but also high-value.

A few caveats though:

It can quickly become "The 5 Whos". You still want to remain blameless.
It assumes that there's a single chain of cause and effect leading to the defect (and there rarely is).

This second reason is the reason I don't really care for The 5 Whys practice anymore. John Allspaw's got some great further discussion about that problem here as well.

Decide on Next Actions as a Team

One common yet major mistake I've seen is that a manager hears the description of the defect/outage and says "well we'll just do x then" without soliciting the input of the team. This is usually a huge mistake because the manager usually doesn't have the intimate knowledge of the system that the developers do, and even if he/she did, it is extremely unlikely that any single person can consistently decide a better course of action than a team. Also, the more a manager does this, the less likely the team is to feel that it's their place to try to solve these problems.

With that said, your team should still be conscious that some of the causal factors are not technical at all. Some examples that I've personally seen are:

The team is over-worked.
The team has been under too much pressure to move quickly or to meet unrealistic deadlines.
The team wants more training in certain areas.
Management is constantly changing priorities and forcing work to be left partially finished.
The team's workflow is too complicated and merges keep leading to undetected defects.
Management won't allow time to be spent on quality measures.

It's a common mistake for developers to focus only on the technical problems, probably because they're the most easily controlled by the development team, but I would say for a team to be truly effective, it must be able to address the non-technical factors as well, and often manage up. Great management will pay very close attention to the team's conclusions.

Resist "Try Harder Next Time"

Hand-in-hand with blamelessness should almost always be a rule that no improvement should involve "Trying harder next time". That would be assuming someone didn't try hard enough the last time, and it's assuming that only effort needs to change in order for the team to be more effective next time. People will either naturally want to try harder next time, or they won't. Saying "try harder next time" usually won't change a thing.

In fact you'd usually be more successful, not just by not trying solutions that don't require more human discipline, but to additionally take that one step further and reduce the level of discipline already required. There's a great blog post on this by Marco Ament here, and I can tell you, the results in real life are often amazing.

Humans are great at conserving their energy for important things, or things that are likely to cause issues, but the result of that is that unlikely events are often not given much effort at all. This is a common trade-off in all of nature that Erik Hollnagel calls the ETTO (Efficiency Thoroughness Trade-Off) principle. You don't want your solutions to be fighting an uphill battle against nature.

There's another kind of strange result of this (I think, anyway) called "the bystander effect", where often if a problem is the responsibility of multiple people, it's less likely that any single person will take responsibility for it. This is a real phenomenon and if you've worked on a team for any length of time, you've seen it happen. You'll want to try to make sure that whatever solutions you come up with, they won't fall victim to this bystander effect.

Consider Cost-to-fix vs. Cost-to-endure

It should go without saying that the cost of your solutions for issues should be less than the cost of the issue itself. Sometimes these costs are really hard to estimate given the number of variables and unknowns involved, but it's at least worth consideration. An early-stage start-up is unlikely to care about being multi-data-center for the sake of redundancy, for instance. It would be ridiculous, on the other hand, for a bank to not seek this level of redundancy.

Consider Process Costs

The second most naive reaction to a bug or outage (after "Try harder next time") is also usually to add some more human process to the team's existing process, like more checks, or more over-sight. While you may well conclude that this is the best measure for a given issue, keep in mind that it's probably also the slowest, most expensive and most error-prone. These sorts of solutions are often the ones that are the simplest to conceive, but if they're your team's only reaction to issues, they will build up more and more over time dooming the team to move really slowly while it works its way repeatedly through these processes.

Improve Defect Prevention

There are a tonne of possible ways to try to prevent bugs from reaching production that are too numerous to get into here, but there are two really important ways to evaluate them:

(1) Does the method find bugs quickly and early in development?

The shortness of your feedback loop in detecting bugs is hugely important in making sure that your prevention methods don't slow the team down so much that their cost outweighs their benefit. Manual human testing by a separate testing team is probably the slowest and most late way to find bugs, whereas syntax highlighters may be both the fastest and earliest for the class of issue that they can uncover (Of course these methods each test completely different things, but they're mentioned to give an idea of both extremes of feedback loops).

(2) Does the method give you enough information to fix the problem quickly/easily?

This criteria is important though admittedly probably less so than the previous one. You will want to judge your prevention measures on this criteria though, because it's another criteria that can cost you a lot of time/efficiency. Incidentally, manual human testing is probably the worst by this criteria as well, because testing at that level generally just lets you know that something is broken in a particular area. Unit-testing beats integration-testing in this particular area as well, because unit-testing does a better job of helping you pin-point the issue down to a particular unit (though they don't actually test for the same types of bugs at all, so it's a bit of an unfair comparison).

With these two criteria in mind, it's useful to look at a number of defect prevention measures critically : TDD, unit testing, integration testing, manual testing, beta testing, fuzz-testing, mutation testing , staging environments, dog-fooding, automated screen-shot diffing, static analysis, static typing, linting, pair programming, code reviews, formal proofs, 3rd-party auditing, checklists etc. I've tried to be a bit exhaustive in that list, and while I've added some options that have never been useful to me, I've probably also forgotten a few. There are new preventative measures popping up all the time too.

An amazing example of preventative measures is probably the extremely popular SQLite project. Their discussion of the measures that they take is fascinating.

Remove Human Processes with Automation

So I've hinted at this a few times so far, but it should be reiterated that automating otherwise manual human processes can bring a level of speed and consistency to them that humans can't compete with. Often times this is tricky and it's not worth it, but these scenarios are getting fewer as technology progresses. There are two huge risks in automation though:

(1) Automation also involves software that can fail. Now you have two systems to maintain and try to keep defect-free.

Often this second system (the automation of the primary system) doesn't get engineered with the same rigor as the primary system, so it's possible to automate in a way that is less consistent and more error-prone than a human.

(2) If you're really going to replace a manual human task, make sure the automation really does outperform the human.

I've seen many attempts at automation not actually meet the goal of doing the job as well as a human. It's not uncommon at all to see examples like teams with 90+% automated test coverage releasing drop-everything defects as awesome as "the customer can't log in" because some CSS issue makes the login form hidden. A team that sees bugs like this often is almost certainly not ready to remove humans from the testing process, regardless of how many tests they've written.

Eliminate Classes of Bugs

When you think about preventing defects without succumbing to "Try Harder Next Time" thought-patterns, one of the most powerful tools is to try to consider how you could make that defect impossible in the future. Often it's possible to avoid defects by working at levels of better abstraction. Here are a few examples:

Avoid off-by-one errors in for loops by using iterator functions, or functional paradigms that give you filter(), map() and reduce().
Avoid SQL injection by using prepared statements.
Avoid Cross-site scripting attacks by using an html template language that automatically escapes output.
Avoid bugs from unexpected changes in shared objects by using immutable data structures.

You may need some creativity here for your own particular defects, but in many cases eliminating the opportunity for a bug to arise is better than trying to catch it when it happens.

For example, I once worked on a team that would occasionally forget to remove console.log() calls from our client-side javascript, which would break the entire site in IE 8. By putting a check for console.log() calls in the build (and breaking it when they exist), we eliminated this class of defect entirely.

Go Beyond Prevention

Defect and outage prevention is only one side of the quality coin as well, though it's usually the side people naturally try to employ when trying to figure out how to handle defects better in the future. You should of course investigate better prevention measures, but you should also consider solutions that will improve your situation when defects do occur, because failures will always happen.

I personally think it's entirely unrealistic for all your measures to be preventative. A focus entirely on preventative measures has a tendency to slow down your team's ability to deliver while at the same time not delivering the level of quality that you could.

With that said, here are a few classes of mitigating measures:

Improve your "Time to Repair"

There's an interesting metric called MTTR which stands for "Mean Time to Repair/Recovery", and is basically the average time it takes you to fix a defect/outage. It's an important metric, because the cost of a defect must include how long that defect was affecting customers. The speed at which you can deliver a fix is going to be a major factor in how defect mitigation. You'll want to ask questions like:

How can we pinpoint problems faster?
How can we create fixes faster?
How can we verify our fixes faster?
How can we deliver fixes faster?

Practices like Continuous Delivery can help here greatly. If you have a 20 minute manual deployment that involves a number of coordinated activities from a number of team members, you will be leaving customers exposed for much longer than a team practicing Continuous Delivery.

Automated testing on its own can be a huge help. If the bulk of your tests is manual, then a fix will take some time to verify (including verifying that it doesn't break anything else). Teams that rely heavily on manual testing will usually test much less thoroughly on a "hot fix", which occasionally can lead to worsening of the situation.

In my experience though, nothing affects MTTR as much as the speed at which you can detect defects/outages...

Improve Detection Time

Talk about how quickly you discovered the issue compared to when it was likely to have started. If your customers are discovering your issues, try figuring out if there's a way that you can beat them to it. Instrumentation (metrics & logging) has been a huge help for me in different organizations for knowing about problems before customers can report them. Information radiators can help keep those metrics ever-present and always on the minds of the team members.

Threshold-based alerting systems (that proactively reach out to the team to tell them about issues) in particular are valuable because they don't rely on the team to check metrics themselves, and they can short circuit that "human polling loop" and alert the team much faster, or during times that they ordinarily would not be looking (at night, on weekends, etc). It's pretty easy to see that an alerting system that alerts about an outage on a Friday night can save the customer days of exposure.

Lessen the Impact of Failures

If you can figure out ways for failures to have less impact, that's a huge win as well. Here are a few examples of ideas I've seen come out of one of these meetings:

Have deployments go to only a 5% segment of users first for monitoring before going out to 100% of the users.
Speed up builds and deployments, so hot-fixes can go out faster.
Have an easy way to display a message to users during an outage
Improve metrics and logging to speed-up debugging around that particular issue.
Set-up off-hours alerting.
Have a one-click "revert deployment" mechanism that can instantly revert to a previous deployment in case something goes wrong.
Create "bulkheads/partitions " in your applications so that if one part fails, the rest can still function properly. There are many common examples of this in software, including PHP's request partitioning model, or the browser's ability to continue despite a javascript exception, even on the same page. Service-oriented architectures often have this quality as well.

You may or may not need some creativity here to come up with your own, but it's worth the effort.

Be Realistic with Plans For Improvement

Whatever you say you will do as a result of this meeting, make sure that it's actually realistic, and there's a realistic plan to get it into the teams future work (eg Who will do it? When?). The best way to have completely useless meetings is to not actually do what you plan to do.

Write Up a Report for the Rest of the Company

The report should say honestly how bad the problem was, in what way (and for how long) customers were affected, and generally what events lead up to it (blamelessly!). Additionally you'll want to declare the next steps that the team plans to take, so that the company knows you're a professional engineering team that cares about results as much as the other people in the organization. You should be ready and willing to take questions and comments on this report, and send it to as many interested parties as possible. Often other people in the company will have additional ideas or information, and the transparency makes them feel like these are welcome any time.

The real value in this is that you show that the entire team is uniformly holding itself accountable for the problem, and that any propensity that the rest of organization has for blaming a single person is not in accordance with the engineering team's views. The engineering team and management should be willing and able to defend any individuals targeted for blame.

Decide What Types of Defects/Outages Necessitate These Meetings

Some organizations are more meeting-tolerant than others, so there's no hard-fast rule here. If you had one of these meetings for every production defect, you'd probably very quickly have a bunch of solutions in place that greatly reduces the number of defects though (and therefore reduces the number of these meetings!). These meetings are all investments. The more you have, the more they start to pay off, both with quality of product and with speed of delivery (if you stay conscious of that!).

One thing I will recommend though is that you look for recurrences and patterns in these defects/outages. The team will usually benefit disproportionately from more time invested in solving repeated problems.

Continuous Testing

2015-01-30T04:40:51Z

Last week I had the pleasure of being on a panel about Continuous Testing put on by Electric Cloud. There's video of the discussion if you're interested, here.

Additionally, I posted a blog post on the subject on the ClassDojo Engineering blog here.

For posterity, I'll x-post here as well:

Continuous Testing at ClassDojo

We have thousands of tests and regularly deploy to production multiple times per day. This article is about all the crazy things we do to make that possible.

How we test our API

Our web API runs on node.js, but a lot of what we do should be applicable to other platforms.

On our API alone, we have ~2000 tests.

We have some wacky automated testing strategies:

We actually integration test against a database.
We actually do api tests via http and bring up a new server for each test and tear it down after.
We mock extremely minimally, because there's little to gain performance-wise in our case, and we want to ensure that integrations work.
When we do more unit-type tests, it's for the sake of convenience of testing a complicated detail of some component, and not for performance.

All ~2000 tests run in under 3 minutes.

It's also common for us to run the entire suite dozens of times per day.

With that said, I usually do TDD when a defect is found, because I want to make sure that the missing test actually exposes the bug before I kill the bug. It's much easier to do TDD at that point when the structure of the code is pretty unlikely to need significant changes.

We aim for ~100% coverage: We find holes in test coverage with the istanbul code coverage tool, and we try to close those holes. We've got higher than 90% coverage across code written in the last 1.5 years. Ideally we go for 100% coverage, but practically we fall a bit short of that. There are some very edge-case error-scenarios that we don't bother testing because testing them is extremely time-consuming, and they're very unlikely to occur. This is a trade-off that we talked about as a team and decided to accept.

We work at higher levels of abstraction: We keep our tests as DRY as possible by extracting a bunch of test helpers that we write along the way, including our api testing library verity.

Speeding up Builds

It's critical that builds are really fast, because they're the bottleneck in the feedback cycle of our development and we frequently run a dozen builds per day. Faster builds mean faster development.

We use a linter: We use jshint before testing, because node.js is a dynamic language. This finds common basic errors quickly before we bother with running the test suite.

We cache package repository contents locally: Loading packages for a build from a remote package repository was identified as a bottleneck. You could set up your own npm repository, or just use a cache like npm_lazy. We just use npm_lazy because it's simple and works. We recently created weezer to cache compiled packages better as well for an ultra-fast npm install when dependencies have not changed.

We measured the real bottlenecks in our tests:

Loading fixtures into the database is our primary bottleneck, so we load fixtures only once for all read-only tests of a given class. It's probably not a huge surprise to people that this is our bottleneck given that we don't mock-out database interaction, but we consider this an acceptable bottleneck given that the tests still run extremely fast.
We used to use one of Amazon EC2's C3 compute cluster instances for jenkins, but switched to a much beefier dedicated hosting server when my laptop could inexplicably run builds consistently faster than EC2 by 5 minutes.

We fail fast: We run full integration tests first so that the build fails faster. Developers can then run the entire suite locally if they want to.

We notify fast: Faster notification of failures/defects leads to diminished impact. We want developers to be notified ASAP of a failure so:

We bail on the build at the first error, rather than running the entire build and reporting all errors.
We have large video displays showing the test failure.
We have a gong sound effect when it happens.

We don't parallelize tests: We experimented with test suite parallelization, but it's really complicated on local machines to stop contention on shared resources (the database, network ports) because our tests use the database and network. After getting that to work, we found very little performance improvement, so we just reverted it for simplicity.

Testing Continues after Deployment

We run post-deployment tests: We run basic smoke tests post-deployment as well to verify that the deployment went as expected. Some are manually executed by humans via a service called Rainforest. Any of those can and should be automated in the future though.

We use logging/metrics/alerts: We consider logging and metrics to be part of on-going verification of the production system, so it should also be considered as part of our "test" infrastructure.

Our Testing Process

We do not have a testing department or role. I personally feel that that role is detrimental to developer ownership of testing, and makes continuous deployment extremely difficult/impossible because of how long manual testing takes. I also personally feel that QA personnel are much better utilized elsewhere (doing actual Quality Assurance, and not testing).

Continuous Testing requires Continuous Integration. Continuous Integration doesn't work with Feature Branches. Now we all work on master (more on that here!).

We no longer use our staging environment. With the way that we test via http, integrated with a database, there isn't really anything else that a staging server is useful for, testing-wise. The api is deployed straight to a small percentage ~5% of production users once it is built. We can continue testing there, or just watch logs/metrics and decide whether or not to deploy to the remaining ~95% or to revert, both with just the click of a button.

We deploy our frontends against the production api in non-user facing environments, so they can be manually tested against the most currently deployed api.

When we find a defect, we hold blameless post-mortems. We look for ways to eliminate that entire class of defect without taking measures like "more manual testing", "more process", or "trying harder". Ideally solutions are automatable, including more automated tests. When trying to write a test for a defect we try as hard as possible to write the test first, so that we can be sure that the test correctly exposes the defect. In that way, we test the test.

Lessons learned:

Our testing practices might not be the best practices for another team. Teams should decide together what their own practices should be, and continually refine them based on real-world results.
Find your actual bottleneck and concentrate on that instead of just doing what everyone else does.
Hardware solutions are probably cheaper than software solutions.
Humans are inconsistent and less comprehensive than automated tests, but more importantly they're too slow to make CD possible.

Remaining gaps:

We're not that great at client/ui testing (mobile and web), mostly due to lack of effort in that area, but we also don't have a non-human method of validating styling etc. We'll be investigating ways to automate screenshots and to look for diffs, so a human can validate styling changes at a glance. We need to continue to press for frontend coverage either way.
We don't test our operational/deployment code well enough. There's no reason that it can't be tested as well.
We have a really hard time with thresholds and anomaly detection and alerts on production metrics such that we basically have to watch a metrics display all day to see if things are going wrong.

The vast majority of our defects/outages come from these three gaps.

I Don't Miss The Sprint

2014-12-02T19:00:23Z

It's been over a year since I've been on a team that does Sprints as part of its engineering process, and I've really changed my mind on their usefulness. I'm probably pretty old-school "agile" by most standards, so I still think Scrum is better than what most teams out there are doing, but I think it's lost me as a proponent for a few reasons, with the most surprising reason (to me) being its use of sprints.

The sprint, for those few engineers that might not have ever developed this way, is simply a time box where you plan your work at the beginning, usually estimating what work will fit within that time box, and then do some sort of review afterward to assess is efficacy, including the work completed, and the accuracy of the estimates.

I've been involved in sprints as short as a week and as long as a month, depending on the organization. The short timespan can have an amazingly transformative effect on organizations that are releasing software less often than that: it gets them meeting regularly to re-plan, it gets them improving their process regularly, and it gets a weekly stream of software released so that the rest of the company can get a sense of what is happening and at what pace. These are huge wins: You get regular course-correction, improvement and transparency, all helping to deliver more working software, more reliably. What's not to love?

Well I've always ignored a couple of major complaints about Sprints because I've felt that those complaints generally misunderstood the concept of the time box. The first complaint is that a Sprint is just another deadline, and one that looms at a more frequent pace so it just adds more pressure. If your scrummaster or manager treats them like weekly deadlines, he/she is just plain "doing it wrong". Time boxes are just meant to be a regular re-visiting and re-assessment of what's currently going on. It's supposed to be a realistic, sustainable pace so that when observed, the speed that the team is delivering at can start to be relied upon for future projections. Treating it like a deadline has the opposite effect:

Developers work at unsustainable paces, inherently meaning that at some point in the future, they will fail to maintain that pace and generally will not be as predictable to the business as they otherwise could be.
Developers cut corners, leading to defects, which lead to future interruptions, which leads to the team slowing more and more over time. I've been on a team that pretty much only had time for its own defects, so I've seen this taken to the absolute extreme.

Here's the thing though, if it's supposed to encourage a reliable, sustainable pace, why would you ever call it a "sprint"? "What's in a name?", right? Well it turns out that when you want to get people to change the way they work, and you want them to understand the completely foreign concepts you're bringing to them, it's absolutely crucial that you name the thing in a way that also explains what it is not.

In Scrum, it's also common to have a "sprint commitment" where the team "commits" to a body of work to accomplish in that time frame. The commitment is meant to be a rough estimate for the sake of planning purposes, and if a team doesn't get that work done in that time, it tries to learn from the estimate and be more realistic in the next sprint. Developers are not supposed to be chastized for not meeting the sprint commitment -- it's just an extra piece of information to improve upon and to use for future planning. Obviously naming is hugely important here too, because in every other use of the word, a "commitment" is a pledge or a binding agreement, and this misnomer really influences the way people (mis)understand the concept of sprints. Let's face it: if people see sprints as just more frequent deadlines (including those implementing them), the fault can't be entirely theirs.

Sooo... The naming is terrible, but the concept is a good one, right?

Well "iteration" is definitely a much better name, and I hope people use that name more and more.

More importantly though, I'd like to argue that time boxes for planning and delivery are fundamentally flawed.

I've personally found that sprint commitments are entirely counter-productive: A team can just gauge its speed for planning purposes based on past performance instead, which is really what the sprint commitment is meant to be. We should be forecasting entirely based on past performance, and adjusting constantly rather than trying to predict, and trying to hold people to predictions.

Also, with planning happening at the beginning of the sprint and software delivery happening at the end, in a regular cadence, the team is much less likely to do things more frequently than this schedule.

Instead of planning and releasing on the schedule of a sprint, we've had a lot more success practicing just-in-time planning and continuous delivery.

Just-In-Time Planning

Planning should happen on-demand, not on a schedule. Planning on a schedule often has you planning for things too far ahead, forcing people to try to remember the plan later. I've seen an hour's worth of planning fall apart as soon as we tried to execute the plan, so I really don't see the point of planning for particular time-spans (especially larger ones). Teams should simply plan for the work they're currently starting, the natural way, not based on time, but based on the work they have to do. They should plan and re-plan a particular task as frequently as is necessary to deliver it, and they should plan for as long or as short as necessary to come up with a plan that everyone is happy with. Planning naturally like this leads to more frequent situations where developers will say "hold on, we should sit down and talk about this before we continue the current plan", which is good for everybody involved.

Continuous Delivery

The cadence of delivering software at the end of a sprint almost always means an organization does not deploy the software more often than that cadence. I've worked on teams that tried to make "deployment day" somewhere in the middle of the sprint to emphasize that the sprint boundary is just an arbitrary time box, decoupled from deployment, but we never managed to deploy more often than once per sprint, and it really only resulted in a de facto change in the sprint schedule for our QA personnel. The very existence of a time-box puts people in the mindset that releasing software, deploying software and calling software "done" are all the same event and there's a scheduled time for that. Without the time-box, people start to think more freely about ways to decouple these events and how to safely improve the customer experience more often. Today, free from sprints, I'm safely deploying to production multiple times per day.

Now Scrum proponents will argue that Sprints can be exactly like this -- sprints do not preclude JIT planning or continuous delivery -- and the time-box just adds additional points of planning and review to ensure that the team is doing the necessary planning and reviewing. While I wholeheartedly agree with this on a theoretical level, this is not what ends up happening in practice: In reality, when you schedule time for particular aspects of work, people tend to wait until that schedule to do that type of work.

And I realize that what I'm suggesting sounds a bit more sloppy, and lacks the formality of a scheduled cadence, but it simply ends up being more natural and more efficient. These concepts aren't new either -- any teams that have been using the venerable kanban method will be familiar with them.

One additional Caveat

With all this said, if a team is going to continually improve, I haven't seen a better mechanism than the regular retrospective. I just don't think Sprints are necessary to make that happen.

Production-Quality Node.js Web Apps : Part III, Preventing Defects

2014-06-23T05:56:10Z

This is the last part of my 3-part series on production-quality node.js web applications. I've decided to leave error-prevention to the end, because while it's super-important, it's often the only strategy developers employ to ensure customers have the most defect-free experience as possible. It's definitely worth checking out part I and part II if you're curious about other strategies and really interested in getting significant results.

Error Prevention

With that said, let's talk about a few error prevention tricks and tactics that I've found extremely valuable. There are a bunch of code conventions and practices that can help immensely, and here's where I'll get into those. With that said I should add that I'm not going to talk about node-fibers, or harmony-generators, or streamline.js, which all have different solutions - ultimately because I haven't used them (which is because none of them are considered very 'standard' (yet)). There are people using them to solve a few of the issues I'll talk about though.

High test coverage, extensive testing

There should be no surprise here, but I think test coverage is absolutely essential in any dynamically typed app because so many errors can only happen at runtime. If you haven't got heavy test coverage, (even coverage itself is valuable!), you will run into all kinds of problems as the codebase gets larger and more complex, and you'll be too scared to make sweeping changes (like change out your database abstraction, or how sessions are handled). An untested/uncovered codebase will have you constantly shipping codepaths to production that have never been actually executed before anywhere.

The simplest way I've found to get code coverage stats (and I've used multiple methods in the past) is to use istanbul. Here's istanbul's output on a file from one of my npm libraries:

The places marked in red are the places that my tests are not testing at all. I use this tool all the time to see what I might have missed testing-wise, when I haven't been doing TDD, and then I try to close up the gaps.

Use a consistent error passing interface and strategy

Asynchronous code should always take a callback that expects an Error object (or null) in the first parameter (the standard node.js convention). This means:

Never create an asynchronous function that doesn't take a callback, no matter how little you care about the result.
Never create an asynchronous function that doesn't take an error parameter as the first parameter in its callback, even if you don't have a reason for it immediately.
Always use an Error object for errors, and not a string, because they have very handy stacktraces.

Following these rules will make things much easier to debug later when things go wrong in production.

Of course there are some cases where you can get away with breaking these rules, but even in those cases, over time, as you use these functions more often and change their internal workings, you'll often be glad that you followed these rules.

Use a consistent error-checking and handling strategy.

Always check the errors from a callback. If you really don't care about an error (which should probably be rare), do something like this to make it obvious:

getUser(userId, function(err, user){
  if (err){
    // do nothing.  we didn't need a user that badly
  }
});

Even then, it's probably best to log the error at the very least. It's extremely rare that you'd bother writing code of which you don't care about the success/failure.

In many cases you won't be expecting an error, but you'll want to know about it. In those cases, you can do something like this to at least get it into your logs:

getUser(userId, function(err, user){
  if (err){
    console.error('unexpected error', err, err.stack);
    // TODO actually handle the error though!
  }
});

You're going to want to actually handle the unexpected error there too of course.

If you're in some nested callback, you can just pass it up to a higher callback for that callback to handle:

getUser(userId, function(err, user){
  if (err){
    return cb(err);
  }
});

NB: I almost always use return when calling callbacks to end execution of the current function right there and to save myself from having to manage a bunch of if..else scenarios. Early returns are just so much simpler in general that I rarely find myself using else anywhere anymore. This also saves you from becoming another "can't set headers after they are sent" casualty.

If you're at the top of a web request and you're dealing with an error that you didn't expect, you should send a 500 status code, because that's the appropriate code for errors that you didn't write appropriate handling for.

getUser(userId, function(err, user){
  if (err){
    console.error('unexpected error', err, err.stack);
    res.writeHead(500);
    res.end("Internal Server Error");
  }
});

In general, you should always be handling or passing up every single error that could possibly occur. This will make surfacing a particular error much easier. Heavy production use has an uncanny ability to find itself in pretty much any codepath you can write.

Watch out for `error` events.

Any EventEmitters in your code (including streams) that emit an error event absolutely need to have a listener for those error events. Uncaught error events will bring down your entire process, and usually end up being the main cause of process crashes in projects that I've worked on.

The next question will be how you handle those errors, and you'll have to decide on a case-by-case basis. Usually you'll at least want to 500 on those as well, and probably log the issue.

Managing Errors at Runtime

Domains

Domains are without a doubt your best tool for catching errors in runtime that you missed at development time. There are three places that I've used them to great effect:

To wrap the main process. When it dies, a domain catches the error that caused it.
To wrap cluster's child processes. If you use cluster-master like I do, or if you use cluster.setupMaster() with a different file specified via exec for child processes, you'll want the contents of the file to be wrapped in a domain as well. This means that when a child process has an uncaught error, this domain catch it.
To wrap the request and response objects of each http request. This makes it much more rare that any particular request will take down your process. I just use node-domain-middleware to do this for me (This seems like it should work in any connect-compatible server as well, despite the name).

In the case of catching an error with a domain on the main process, or a child process, you should usually just log the issue (and bump a restart metric), and let the process restart (via a process manager for the main process, or via cluster for the child process -- see part I for details about this). You probably do not want to the process to carry on, because if you knew enough about this type of error to catch it, you should have caught it before it bubble up to the process level. Since this is an unexpected error, it could have unexpected consequences on the state of your application. It's much cleaner and safer to just let the process restart.

If you've caught a an error from request or response object, it's probably safe and appropriate to send a 500 status code, log the error (and stack trace) and bump a 500 metric.

Also, I've written a module that helps me test that my domains are correctly in place: error-monkey. You can temporarily insert error-monkey objects in different places in your codebase to ensure that domains are catching them correctly. This way, you're not tweaking the location of your domains and waiting for errors to happen in production to verify them.

Gracefully restarting

Now that domains have given you a place to catch otherwise uncaught exceptions and you can hook in your own restart logic, you'll want to think about gracefully restarting rather than just killing the process abruptly and dropping all outstanding requests in progress. Instead you'll want tp restart only after all current requests have completed ( see : http://nodejs.org/api/http.html#http_server_close_callback ). This way, the process can be restarted more gracefully.

If you're running a bunch of node processes (usually with cluster), and a number of servers (usually behind a load-balancer), it shouldn't be that catastrophic to close one instance temporarily, as long as it's done in a graceful manner.

Additionally, you may want to force a restart in the case that the graceful shutdown is taking an inordinate amount of time (probably due to hanging connections). The right amount of time probably has a lot to do with what types of requests you're fulfilling, and you'll want to think about what makes sense for your application.

With that said...

These are just my notes on what I've tried so far that achieved worthy results. I'd be interested to hear if anyone else has tried any other approaches for preventing, mitigating, and detecting defects.

Production-Quality Node.js Web Apps : Part II, Detecting Defects

2014-06-03T00:41:31Z

This is the second part of a three part series on making node.js web apps that are (what I would consider to be) production quality. The first part really just covered the basics, so if you're missing some aspect covered in the basics, you're likely going to have some issues with the approaches discussed here.

First, I'm going to talk about a few major classes of defects and how to detect them.

I could go straight to talking about preventing/avoiding them, but detection is really the first step. You're simply not going to get a low-defect rate by just using prevention techniques without also using detection techniques. Try to avoid the junior engineer mindset of assuming that "being really careful" is enough to achieve a low defect rate. You can get a much lower defect rate and move much faster by putting systems in place where your running application is actively telling you about defects that it encounters. This really is a case where working smart pays much higher dividends than working hard.

The types of defects you're most likely to encounter

I'm a huge fan of constantly measuring the quality aspects that are measurable. You're not going to detect all possible bugs this way, but you will find a lot of them, and detection this way is a lot more comprehensive than manual testing and a lot faster than waiting for customers to complain.

There are (at least) 4 types of error scenarios that you're going to want to eliminate:

restarts
time-outs
500s
logged errors

Restarts

Assuming you've got a service manager to restart your application when it fails and a cluster manager to restart a chlid process when one fails (like I talked about in Part I), one of the worst types of error-scenarios that you're likely to encounter will be when a child process dies and has to restart. It's a fairly common mistke to believe that once you have automatic restarting working, process failure is no longer a problem. That's really only the beginning of a solution to the problem though. Here's why:

Node.js is built to solve the C10K problem, so you should expect to have a high number of requests per process in progress at any given time.
The node process itself handles your web-serving, and there's no built-in request isolation like you might find in other frameworks like PHP, where one request can't have any direct effects on another request. With a node.js web application, any uncaught exception or unhandled error event will bring down the entire server, including your thousands of connections in progress.
Node.js's asynchronous nature and multiple ways to express errors (exceptions, error events, and callback parameters) make it difficult to anticipate or catch errors more holistically as your codebase gets larger, more complex, and has more external or 3rd-party dependencies.

These three factors combine to make restarts both deadly and difficult to avoid unless you're very careful.

Detection: The easiest way to detect these is to put metrics and logging at the earliest place in both your cluster child code and your cluster master code to tell you when the master or child processes start. If you want to remove the noise caused by new servers starting up, or normal deployments, then you may want to write something a little more complex that can specifically detect abnormal restarts (I've got a tiny utility for that called restart-o-meter too).

You might have an aggregated logging solution than can send you alerts based on substrings that it finds in the logs too, or that can feed directly into a time-series metrics system.

Time-outs

Time-outs are another type of failed request, where the server didn't respond within some threshold that you define. This is pretty common if you forget to call res.end(), or a response just takes too long.

Detection: You can just write a quick and dirty middleware like this to detect them:

app.use(function(req, res, next){
  timeoutThreshold = 10000; // 10 seconds
  res.timeoutCheck = setTimeout(function(){
    metrics.increment("slowRequest");
    console.error("slow request: ", req.method.toUpperCase(), req.url);
  }, timeoutThreshold);
  res.on('finish', function(evt){
      clearTimeout(res.timeoutCheck);
  });
  next();
});

(or you can grab this middleware I wrote that does basically the same thing).

500s

On a web application, one of the first things I want to ensure is that we're using proper status codes. This isn't just HTTP nerd talk (though no one will say I'm innocent of that), but rather a great system of easily categorizing the nature of your traffic going through your site. I've written about the common anti-patterns here, but the gist of it is:

Respond with 500-level errors if the problem was a bug in your app, and not in the client.

This lets you know that there was a problem, and it's your fault. Most likely you only ever need to know 500 Internal Server Error to accomplish this.

Respond with 400-level errors if the problem was a bug in the client.

This lets you know that there was a problem and it was the client app's fault. If you learn these 4 error codes, you'll have 90% of the cases covered:

400 Bad Request -- When it's a client error, and you don't have a better code than this, use this. You can always give more detail in the response body.
401 Not Authenticated -- When they need to be logged in, but aren't.
403 Forbidden -- When they're not allowed to do do something
404 Not Found -- When a url doesn't point to an existing thing.

If you don't use these status codes properly, you won't be able to distinguish between:

successes and errors
errors that are the server's responsibility and errors that are the client's responsibility.

These distinctions are dead-simple to make and massively important for determining if errors are occuring, and if so, where to start debugging.

If you're in the context of a request and you know to expect exceptions or error events from a specific piece of code, but don't know/care exactly what type of exception, you're probably going to want to log it with console.error() and respond with a 500, indicating to the user that there's a problem with your service. Here are a couple of common scenarios:

the database is down, and this request needs it
some unexpected error is caught
some api for sending email couldn't connect to the smtp service
etc, etc

These are all legitimate 500 scenarios that tell the user "hey the problem is on our end, and there's no problem with your request. You may be able to retry the exact same request later". A number of the "unexpected errors" that you might catch though will indicate that your user actually did send some sort of incorrect request. In that case, you want to respond with a reasonable 4xx error instead (often just a 400 for "Bad Request") that tells them what they did wrong.

Either way, you generally don't want 500s at all. I get rid of them by fixing the issues that caused them, or turning them into 4xxs where appropriate (eg. when bad user input is causing the 500), to tell the client that the problem is on their end. The only times that I don't try to change a response from a 500 is when it really is some kind of internal server error (like the database is down), and not some programming error on my part or a bad client request.

Detection: 500s are important enough that you're going to want to have a unified solution for them. There should be some simple function that you can call in all cases where you plan to respond with a 500 that will log your 500, the error that caused it, and the .stack property of that error, as well as incrementing a metric, like so:

var internalServerError = function(err, req, res, body){
    console.log("500 ", req.method, req.url);
    console.error("Internal Server Error", err, err.stack);
    metrics.increment("internalServerError");
    res.statusCode = 500;
    res.end(body);
};

Having a unified function you can call like this (possibly monkey-patched onto the response object by a middleware, for convenience) gives you the ability to change how 500's are logged and tracked everywhere, which is good, because you'll probably want to tweak it fairly often.

You're probably actually going to use this on most asynchronous functions in your HTTP handling code (controllers?) that you call that you don't expect an error on. Here's an example:

function(req, res){
    db.user.findById(someId, function(err, user){
        if (err) return internalServerError(err, req, res, "Internal Server Error");
        // ...
        // and normal stuff with the user object goes here
        // ...
    });
}

In this case, I just expect to get a user back, and not get any errors (like the database is disconnected, or something), so I just put the 500 handler in the case that an err object is passed, and go back to my happy-path logic. If 500s start to show up there at runtime, I'll be able to decide if they should be converted to a 4xx error, or fixed.

For example if I start seeing errors where err.message is "User Not Found", as if the client requested a non-existant user, I might add 404 handling like so:

function(req, res){
    db.user.findById(someId, function(err, user){
        if (err) {
            if (err.message === "User Not Found"){
                res.status = 404;
                return res.end("User not found!")
            }
            return internalServerError(err, req, res, "Internal Server Error");
        }
        // ...
        // and normal stuff with the user object goes here
        // ...
    });
}

Conversely, if I start seeing errors where err.message is "Database connection lost", which is a valid 500 scenario, I might not add any specific handling for that scenario. Instead, I'd start looking into solving how the database connection is getting lost.

If you're building a JSON API, I've got a great middleware for unified error-handling (and status codes in general) called json-status.

A unified error handler like this leaves you with the ability to expand all internal server error handling later too, when you get additional ideas. For example, We've also added the ability for it to log the requesting user's information, if the user is logged in.

Logged Errors

I often make liberal use of console.error() to log errors and their .stack properties when debugging restarts and 500s, or just reporting different errors that shouldn't necessarily have impact on the http response code (like errors in fire-and-forget function calls where we don't care about the result enough to call the request a failure).

Detection: I ended up adding a method to console called console.flog() (You can name the method whatever you want, instead of 'flog' of course! I'm just weird that way.) that acts just like console.error() (ultimately by calling it with the same arguments), but also increments a "logged error" metric like so:

console.flog = function(){
  if (metrics.increment){
    metrics.increment("loggedErrors");  
    // assume `metrics` is required earlier
  }

  var args = Array.prototype.slice.call(arguments);
  if (process.env.NODE_ENV !== "testing"){
    args.unshift("LOGGED ERROR:");  
    // put a prefix on error logs to make them easier to search for
  }  

  console.error.apply(console, args);
};

With this in place, you can convert all your console.error()s to console.flog()s and your metrics will be able to show you when logged errors are increasing.

It's nice to have it on console because it sort of makes sense there and console is available everywhere without being specifically require()ed. I'm normally against this kind of flagrant monkey-patching, but it's really just too convenient in this case.

Log Levels

I should note too that I don't use traditional log levels (error/debug/info/trace) anymore, because I don't find them all that useful. I'm logging everything as an error or not, and I generally just strive to keep the logs free of everything other than lines that indicate the requests being made like so:

  ...
  POST /api/session 201
  GET /api/status 200
  POST /api/passwordReset 204
  GET /api/admin.php 404
  ...

That doesn't mean that I don't sometimes need debug output, but it's so hard to know what debug output I need in advance that I just add it as needed and redeploy. I've just never found other log-levels to be useful.

More About Metrics

All of the efforts above have been to make defects self-reporting and visible. Like I said in the first part of this series, the greatest supplement to logging that I've found is the time-series metrics graph. Here's how it actually looks (on a live production system) in practice for the different types of defects that have been discussed here.

The way to make metrics most effective, is to keep them on a large video display in the room where everyone is working. It might seem like this shouldn't make a difference if everyone can just go to their own web browsers to see the dashboard whenever they want, but it absolutely has a huge difference in impact. Removing that minor impediment and just having metrics always available at a glance results in people checking them far more often.

Without metrics like this, and a reasonable aggregated logging solution, you are really flying blind, and won't have any idea what's going on in your system from one deployment to the next. I personally can't imagine going back to my old ways that didn't have these types of instrumentation.

Production-Quality Node.js Web Apps : Part I, The Basics

2014-06-01T17:09:00Z

I've been working on production-quality node.js web applications for a couple of years now, and I thought it'd be worth writing down some of the more interesting tricks that I've learned along the way.

I'm mostly going to talk about maintaining a low-defect rate and high availability, rather than get into the details about scaling that are covered in a lot of other places. In particular, I’ll be talking about load-balancing, process management, logging, and metrics, and the how’s and why’s of each.

Balance the Load

I'm going to assume that you're already load-balancing on a given server with cluster or some higher level abstraction ( I use cluster-master) as well as between servers with a load-balancer like ha-proxy.

Performance considerations aside, your service will have much better availability, quality, and uptime if you've got multiple processes running on multiple machines. More specifically, you get:

the ability to immediately failover in the case of single process or single machine failure
reduced overall service failure in the case of single process or single machine failure
the ability to gracefully deploy with the load-balancer

Gracefully Deploy

Gracefully deploying means that your deploy process has enough servers running to handle the load at all times throughout the actual deployment process, and that none of the servers are taken off-line while outstanding requests are still in progress. This means:

Your clustering solution must be able to stop new connections and not exit the master process until all servers have finished processing their existing requests. The solution I personally use is cluster-master, but there are bunch of suitable alternatives.
You need a rolling deployment, meaning that servers are restarted one-at-a-time, or in small groups, rather than all at once. The easiest way to do this is probably to write a nice deploy script that takes restarting servers out of the load-balancer until they're running again.

If you don't have a graceful deploy solution, every deployment of new code will lose requests in progress, and your users will have a terrible experience.

Also note: I've seen a few clustering solutions that use some sort of hot-deploy (hot-deploy loads a new version of code without taking the server down) functionality. If you've got a rolling deploy via your load balancer though, you probably don't need any sort of hot-deploy functionality. I'd personally avoid solutions that involve the complexity of hot-deploying.

Run as a Service

You're also going to want to be running your app as a service that the OS knows to restart with some service manager like upstart. A service manager like this is going to be absolutely essentially for when your node.js app crashes, or when you spin up new machines.

It's probably worth noting that you won't really want to use something like forever or nodemon in production, because it doesn't survive reboots, and is pretty redundant once you've added service management that actually does (This is a case where you don't want redundancy, because these types of process managers can end up fighting with each other to restart the app, thus never really allowing the app to start).

Log to Standard Output

Logging to standard output (using console.log()) and standard error (using console.error()) is the simplest and most powerful way to log. Here's how:

pipe it, don't write it

In the config file for running your app, you want something like this to specify a log file:

node server.js >> /var/log/myserver.log 2>&1

The >> tells your node process to append output to the specified log file and the 2>&1 tells it that both standard out and standard error should go to that same log file. You don't want to be writing to the logs programmatically from within the node process, because you will miss any output that you don't specifically log, like standard error output from node.js itself which happens anytime that your server crashes. That kind of information is too critical to miss.

console is the only "logging library" you need

With this kind of logging set up, I just have to console.log() for any debug output that I need (usually just temporarily for a specific issue that I'm trying to solve), or console.error() for any errors I encounter.

Additionally, one of the first things I do on a new web service is to set up a console.log() for each request (this should work in any express or connect app):

  app.use(function(req, res, next){
    res.on('finish', function(evt){
      console.log(req.method, req.url, res.statusCode);
    });
  });

This chunk of code gives me nice simple logs for every request that look like this:

...
POST /api/session 400
POST /api/session 401
POST /api/session 200
GET /api/status 200
GET /api/status 200
...

rotating the logs

The missing infrastructure needed to support this is a way to rotate the logs, like logrotate. Once that's set up properly, your logs will rotate for you nicely and not fill up your disk on you.

Tools to Help Detect Problems at Runtime

There are two basic key ways that I like to instrument an application to detect problems that occur at runtime: Aggregated logging and metrics.

Agreggated Logging

One of the most important things you can do for error and defect detection is to have aggregated logging -- some service that brings all your web servers' logs together into one large searchable log. There are a few products for this: The stand-out open source one for me seemed to be the logstash/kibana combination, though these days I'm lazy and generally use the papertrail service instead.

I would highly recommend that you set a service like this up immediately, because the small amount of friction involved in sshing into servers to tail logs is enough to seriously reduce how often you and your teammates will actually look at the logs. The sooner you set this up, the sooner you can benefit from being able to learn about your application through the logs that you have it write.

Metrics

When I say "metrics" I really mean time-series metrics, that allow me to see the frequency of different types of events over time. These are invaluable because they

tell you when something unusual is happening
aggregate certain types of data in ways that logs can't
help you rally the team or company around specific high-value goals (be careful with this one!)

The stand-out metrics/graphic open source product is probably graphite. I've generally been using Librato's metrics service though because it's easy to set up, and looks great, so that's where I'll pull my screenshots from for time-series data. I've also had a pretty good experience with DataDog's service as well. Both also come with the ability to raise alerts when a metric surpasses a threshold, which can be a nice way to know when something is going on that you should investigate.

Basic Metrics

There are a bunch of basic metrics that you can track to see what's going on in your server.

Here's an example of some very high-level metrics for us over a week:

in blue: number of requests
in green: average duration of requests

(Note that at this resolution, the y-values are averages over such long durations that the values aren't really that useful at face value; it's the visualization of general trends that is useful.)

There are a few obvious things that should jump out right away in this example:

We have daily traffic spikes with 5 peaks really standing out and 2 being almost flat (those happen to be Saturday and Sunday).
We had a spike in average request duration (on Friday morning -- the third peak). This was caused by some performance issues with our database, and resulted in service outage for a number of our users during that time.

I can basically put metrics.increment("someEventName"); anywhere in my codebase to tell my metrics service when a particular event occurred.

Also consider OS metrics like:

disk space usage
cpu utilization
memory consumption
etc, etc

I've got my codebase set up so that metrics.gauge("someMetricName", value); will allow me to graph specific values like these over time as well.

If you're not already doing monitoring like this, you must be wondering what your servers would tell you. When it's this easy, you tend to do it all the time and get all kinds of interesting metrics including more business-specific metrics.

What next?

These are really just the basics. If you're not doing these things, and you care about running a production-quality web service, you're probably putting yourself at a disadvantage.

There is a lot more that you can do though, and I'll get into that in my next post, which will be Part II to this one.

In Defence Of The Office

2014-01-11T22:51:15Z

I've been reading the 37 signals book Remote lately and it's got me thinking a lot about the advantages and disadvantages of working remotely. While they clearly acknowledge that there are trade-offs to working remotely, I personally don't think they paint a very clear picture of just how expensive those trade-offs are. I can only really speak from a software development perspective, but I think this might apply to other types of businesses where a high degree of collaboration is necessary and the concept of "team" is more than just "a group of people".

After 7 or 8 years of being a home-based contract-worker, and close to the same time spent working in various office environments, I've found that with the right people and the right environment, the office can be a lot more productive (and actually enjoyable) than a home environment, for a number of reasons.

Skype / Google Hangouts / Webex pale in comparison to actually being in the same room as someone else. I can't explain why, because it seems to me like it should be good enough (and I really wish it were sometimes), but the fact of the matter is that it's not. You miss nuances in communication that can add up to big differences in productivity. I've been in hours and hours of pair programming sessions over webex: it doesn't match the high-bandwidth of shoulder-to-shoulder pairing. I've done the shared whiteboard thing: it doesn't match the speed and simplicity of an actual whiteboard.

With communication over today's technological mediums, there's just a natural tendency to end the interaction as soon as possible "and get back to work" too, where people don't actually leave Skype / Google hangouts / webex running all day long. The result is that people are reluctant to start new conversations like that because they're worried about interrupting the other person. So people naturally fall back on less intrusive means of communication (text chat) and often even further back to more asynchronous forms of communication (email).

And asynchronous communication is not the boon to productivity that people think it is. Email is today's definitive means of asynchronous communication, but I think it's pretty obvious that there are few methods of communication that are less efficient. Imagine a sports team that can only communicate by email. That would be ridiculous. Email is amazing at its ability to cut down on interruptions of course, but it's at the obvious cost of timeliness and immediacy (not to mention the very real human aspects including general tone and emotional nuances!).

The ideal solution would be to solve the problem of interruptions without losing timeliness.

Explaining this really requires actually defining an "interruption": When you're on a team and someone wants to talk to you about the main goal of the team, that's decidedly NOT an interruption. Imagine a quarterback being annoyed by the "interruption" of someone yelling that they're open for a pass. Imagine a soldier in a firefight being annoyed at another soldier for requesting cover fire prior to some forward manoeuvring. In order for something to really be an interruption, you have to be working on a different priority than the other person, one that you think is of higher priority.

In a collaborative environment, interruptions should often be viewed as a symptom, and not the actual cause, of the problem. To solve the problem of interruptions at the root, you've got to clearly define the priorities of the team, aligning everyone on those, and concentrating as many people as possible on the top goal rather than going the easy route (management-wise) and doing something like giving each of the X members of the team one of the top X priorities, effectively "team-multi-tasking". Team-multi-tasking is a common approach because:

It's the easiest thing to do management-wise (Why bother prioritizing when I can have X tasks worked on at once?).
It feels like you're getting more done, because so many things are in progress at once.
It ensures everyone is 100% utilized.

But it's also pretty obviously the absolute slowest way to get your top priority shipped and to start getting value from it (and often the slowest way to get them all done, believe it or not!).

Not only that, but the more you do this, the more people tend to specialize into roles like "the back-end guy" or the "ops guy", etc, and individuals lose the ability to be cross-functional, and to practice collective code ownership. It's a vicious cycle too: the more an individual tends toward deep specialization, the more we're tempted to give them that kind of work because they're the best at it. Not only does your bus factor skyrocket, but you get back into these scenarios where anytime someone engages someone else for help, it's an interruption to that other person, so people tend to not ask for help when they really should, or they use dog-slow asynchronous methods (like email). Breaking this cycle means constantly trying to break down these specialty silos for any given specialty by having more experienced people collaborate with (and often mentor) less experienced people. The end result is a team that makes full-stack engineers and not just one that hires them.

I find that the right mindset for the team is to create an environment that would be best for a team working on only 1 super-important thing that you want to start getting value from as soon as possible. A general rule of thumb is: If you're having a text conversation with someone in the same room, you're doing it wrong. I know that's common, but if you think about it, it's pretty absurd.

How would that environment look? It would be a "warroom" for radically collocating the team. Everyone that needs to be there would be there, arranged in a manner that's most effective for verbal communication. There would be nearby whiteboards for collaborative design sessions, and information radiators to show the progress of the efforts and various other metrics of the current state of affairs by just glancing up.

How would the team behave? They would be a "tiger team". They would all be helping to get that one thing out the door. They would almost never use any electronic means to communicate (except obviously, when it's more efficient, like sharing chunks of code, or urls), and you'd never hear someone say "that's not my job". If someone is the only person that knows how to do something, the team identifies them as a process choke-point and quickly tries to get others up to speed on that skill or responsibility (and not by giving them links to docs or occasional hints, but by super-effective methods like pair programming, in-person demonstration, and actual verbal discussion). If one member of the team appears to be stuck, he asks for help, and if he doesn't, the other members notice, and jump in to help, unprompted. There are no heroes, and everyone takes responsibility for everything that the team is tasked with. This can and should be taken to extremes too: members should drop their egos and make themselves available to do grunt work or testing for other members -- whatever gets the top priority shipped as soon as reasonably (and sustainably) possible. This includes making yourself vulnerable enough to ask for help from others, not only for tricky aspects of your work, but also just to divide your work up better. If you're taking longer than expected on a given task, you should be conscious enough of that to be openly discussing it with the team, including possible solutions, and not trying to be a hero or a martyr. Conversely, you should never be waiting for extended periods of time for your teammate to do some work that you depend on for something else. If a teammate is taking longer than usual, jump in and help.

How would management work then? The team should self-organize. With clear priorities set, the team can and should, for the most part, self-direct. A manager should not try to manage the avalanche of interactions that happen during free-collaboration throughout the day. Trying to manage who works on what, will simply make that manager a choke-point and slow the process down (and he/she will inevitably be wrong more than right, compared to the wisdom of the entire team). Often a non-technical manager can have the advantage over more technical managers, because he's forced to trust the team's decisions and doesn't become an "approval choke-point" for the team's engineering decisions.

That doesn't mean a management role is unnecessary though; it's actually quite demanding to manage a team like this. Priorities must always be crystal clear and for the most part, the top few priorities being worked on should not change very often. If they do change often, you usually have either a management problem or a product quality problem. Those problems should be fixed at the root as well, rather than making developers feel like management or customers are just a constant stream of interruptions. Keep in mind that if you do change the top priority often enough, you can completely prevent the team from any progress at all (I've spent months on a team like that before -- it's not fun for anyone). (For a more complete description of the ideal responsibilities for this style of management, see "The Management" here )

Does this work for all types of people? No. But then again, you'll never get a fast team of developers without hiring the right people. If you have people on the team that are incapable of being team-players, it's certainly not going to work (and you'll likely have a number of problems).

What if we have TWO top priorities though? Flip a coin and pick one; It's that simple. Just arbitrarily choosing one will get one of them shipped and returning value as soon as possible.

What about team member X that has nothing to do? The team should recognize this and:

try to break down the tasks better so they can help (it's often worth a 30 minute session to bring other people on board -- Any given task can almost always be broken down further with additional design discussion, even if its not worth dividing up the work).
try to get them pair-programming so they can learn to help better.
try to get them testing the parts that are done.

Obviously in some scenarios it might make sense for them to start on the second priority, but they should be ready to drop that work at any time to help get priority #1 out the door faster. Remember: the goal is to ship finished features faster. It's not to keep the people on the team as busy as possible.

Isn't that really tough on the developers to rarely have quiet time to concentrate? Yes, at first it often is, often because focusing on the top priority requires discipline because it's different from the status quo. And people simply aren't used to the buzz in the room. The longer you spend working with headphones on, trying to carve out your own niche, separate from the rest of the team, the longer it will take you to transition and the harder that transition will be. But soon you learn to ignore the irrelevant buzz in the room and to tune back in quickly when it's important and relevant. And since you're all working on the same thing, it's often relevant, and lot less difficult to get back into the flow state than you'd expect (especially if pair programming). So the pay-off for developers is:

huge productivity improvements
less waiting for someone else to do their part
longer sessions in "flow state" and easier returns to flow state
less solitude without adding interruptions
less "all the weight is on my shoulders" scenarios
more knowledge sharing and personal development

It helps a lot if developers take breaks more often, because you actually end up spending a lot more time in the flow state in a given day. You end up learning more from other developers and being vastly more productive as well, which is exhausting and energizing at the same time. In my opinion, when done right, it's just a lot more fun than isolated development.

Of course it often also helps to have break-out rooms for smaller prolonged conversation when desired as well.

But what if you have dozens of engineers? It's just not reasonable to try to make teams that contain dozens of engineers. Brooks' law has seemed to hold fairly well over time and he explains that one reason is that the more people you add to a group, the more communication overhead there is. I've personally seen this as well, with team-size starting to get diminishing returns between 8 and 12 people, with little value (if any) to adding members beyond that. When you get beyond 8 people on a given team, you need to start thinking about creating a second team. Conway's Law seems to dictate that a service-oriented architecture is the best solution we're going to get for scaling up an organization's developer head-count, but I've personally found that smaller codebases with distinct responsibilities are generally better codebases anyway.

With all that said, of course I really wish I could work remotely sometimes and get all the same benefits. I've simply never seen a telepresence technology set-up that matches the fidelity of actual face-to-face and shoulder-to-shoulder collaboration. Maybe it exists. I'd love to hear about it. But my experience simply doesn't indicate that the current status quo technologies (skype, webex, etc), like Remote recommends, are up to snuff. The way it stands today, I'll almost always put my money on a collocated group of "team-players" over a geographically disparate team of "gurus".

You probably don’t need to version your web API.

2013-09-14T01:47:31Z

So you’re thinking about versioning your web API somehow. Versioning is how we annotate and manage changes in software libraries, so it seems natural to re-apply this solution to changes in web APIs. Most web APIs are versioned in order to…

( A ) “freeze” an older version somehow so that it doesn’t change for older clients, or at least it is subjected to much less change than newer versions, so you can still move forward.

( B ) be able to deprecate older APIs that you no longer wish to support.

( C ) indicate that changes have occurred and represent a large batch of changes at once when communicating with others.

However…

( A ) Versions don’t actually solve the problem of moving forward despite older clients.

You can safely break backward compatibility with or without the concept of versions: just create a new endpoint and do what you want there. It doesn’t need to have any impact on the old endpoint at all. You don’t need the concept of “versions” to do that.

( B ) Versions don’t actually solve the problem of deprecating old endpoints.

At best, versions just defer the problem, while forcing you to pay the price of maintaining two sets of endpoints until you actually work out a real change management strategy. If you have an endpoint that you want to retire, you’re going to have to communicate that to all your clients that you’re deprecating it, regardless of whether or not you’ve “versioned” it.

( C ) Versions DO help you communicate about changes in a bunch of endpoints at once. But you shouldn’t do this anyway.

Of course, if you’re retiring 20 endpoints at once, it’s easier to say “we’re retiring version 1” than to say “we’re retiring the following endpoints…” (and then list all 20), but why would you want to drop 20 breaking-change notices on your clients all at once? Tell your clients as breaking changes come up, and tell them what sort of grace period they get on those changes. This gives you fine-grained control over your entire API, and you don’t need to think about a new version for the entire API every time you have a breaking change. Your users can deal with changes as they occur, and not be surprised with many changes at once (your new version).

“So how the @#$% do I manage change in my API??”

With a little imagination, it’s totally possible. Facebook is one of the largest API providers in the world, and it doesn’t version its API.

Take Preventative measures: Spend more time on design, prototyping, and beta-testing your API before you release it. If you just throw together the quickest thing that could possibly work, you’re going to be paying the price for that for a long time to come. An API is a user interface, and as such it needs to actually be carefully considered.

Consider NOT making non-essential backwards-incompatible changes: If there are changes you’d like to make in order to make your API more aesthetically-pleasing, and you can’t do so in an additive fashion, try to resist those changes entirely. A good public-facing API should value backward compatibility very highly.

Make additive changes when possible: If you want to add a new property to some resource, just add it. Old clients will just ignore it. If you want to make breaking changes to some resource, consider just creating a new resource.

Use sensible defaults where possible: If you want a client to send a new property, make it optional, and give it a sensible default.

Keep a changelog, roadmap and change-schedule for your API: If you want to solve the problem of keeping your users abreast of changes in your API, you need to publish those changes somehow.

Use hypermedia: Tell the clients what uris to use for the requests that it needs to make and give them uri templates instead of forcing them to make their own uris. This will make your clients much more resilient to change because it frees you up to change uris however you want. Clients making their own uris are tomorrow’s version of parsing json with regular expressions.

NB!

Notice that almost all of these are things that you should do regardless of whether or not you version your API, so they’re not really even extra work.

Notice also that I never said “Never make changes”. I’m not trying to make an argument against changes. I’m just saying that versioning doesn’t help you make changes at all. At best it just lets you put a bunch of changes in a “bucket” to dump on your API users all at once.

My Favourite New Ility

2013-06-10T04:21:46Z

If you’ve ever seen a laundry list of the types of non-functional requirements like testability, accessibility, reliability, etc, (often called “ilities” because of their common suffix) you probably haven’t seen “rewritability”, but I really want to add it to the pile.

I’ve often found in software engineering that counterintuitively, if something is difficult or scary there’s a lot to be gained by doing it more often. I think we can probably agree that software rewrites are amoung those difficult and scary tasks.

Now that we’re starting to get better at service-oriented architecture, where networked applications are better divided into disparate services that individually have narrow responsibilities, a natural side effect is that it’s getting easier to write services that are replaceable later.

So what makes a piece of software rewritable?

Clearly defined requirements: not just what it needs to do, but also what it won’t do, so that feature-creep creeps into another more appropriate service. This one ends up requiring some discipline when the best place and the easiest place for a given responsibility don’t end up being the same service, but it’s almost always worth it.

Clearly defined interface: If you want to be able to just drop a new replacement in place of a previous one, it’s got to support all the integration points of the previous one. When interfaces are large and complicated this gets a lot harder and a lot riskier. In my opinion, this makes a lot of applications with GUIs very difficult to rewrite from scratch without anyone noticing (Maybe it’s because UX is hard. Maybe that’s because I’m terrible at GUIs.).

Small: Even with clearly defined requirements and interface, a rewrite is still going to be expensive and hard to estimate if there’s a lot to rewrite.

Any software engineers that have found themselves locked into a monolithic ball of mud application can see the obvious upsides of having a viable escape route though.

And once we’re not scared of rewrites, what new possibilities emerge? Real experimentation? Throw-away prototypes that we actually get to throw away?

How much easier is it to reason about a piece of software when it also meets all the criteria that make it easy to rewrite?

A Requiem for a Team

2013-05-08T04:27:20Z

I once asked an interviewee “Ever work on a high-performing team?” and he answered “Yeah…” and then went on to explain how his team profiled and optimized their code-base all the time to give it very high performance.

That’s not what I meant, but in his defence, I really don’t think many developers have had the opportunity to work on high-performing teams and so they don’t know what they don’t know.

I’ve been developing software professionally for around 15 years and I’ve only worked on a high-performing team once. It was a couple of years ago, and it was awesome. I haven’t been able to find another since, but now I’m a true-believer in the concept of a 10X team.

I suspect that there might be groups out there somewhere who have experience with even higher-performing teams, but I’m certain that I’ve been on enough teams now that high-performing teams are extremely few and far-between. I’m not sure I even know how to create one, but I know one when I see one. I’ve definitely been on teams since that have been close to being high-performing.

The high-performing team is the ultimate win-win in the software development industry because the developers on the team are thrilled to be going into work on a daily basis and the business is thrilled because they’re producing results much faster and more reliably than other teams. Those are really the two essential criteria, in my opinion:

Is everybody happy to be part of the team?
Is the team producing results quickly? The first criteria is absolutely necessary for the second to be sustainable. Any business that neglects developer enjoyment is taking a short-sighted path that will eventually negatively impact performance (due to burnout, employee turnover, etc).

So all of this sounds great, but the real mystery is in how to make it happen. I’m not 100% sure that I know. I don’t think I’ve ever even heard of people talking about how to achieve it. This is my shot at describing the recipe though.

The Management

The craziest part of my experience is that the only time I’ve seen a high performance team happen was when the manager of the team had very little software development experience. That seems maddeningly counter-intuitive to me because from a developer’s perspective, one of the most frustrating aspects of software development is having a clueless person telling you what to do and how to do it. It turns out that being clueless in software development can be a huge benefit to being brilliant at software development management though, if the manager is already brilliant with management in general. This manager was definitely brilliant on the management side. Here’s why:

He hired for attitude over aptitude. He would hire based on peoples’ potential, their attitudes, and their willingness to work hard instead of whether their precise skillset filled a niche that we needed.
He would cull developers who weren’t team players. It’s not a job anyone wants to do, but it’s sometimes necessary to let people go when they’re not working out. Not doing this results in the rest of the team being miserable.
He kept the team small, and didn’t move people in or out of it very often. This gives the team the maximum opportunity to form into a cohesive unit.
He would set clear goals and priorities. No two things would share the same priority (even though I’m sure he was probably secretly flipping a coin when more than one thing was extremely important). That would give the team a clear direction and an ability to focus on what’s most important.
He never explained a requirement with "because I said so". This encouraged a culture of questions and feedback which enhanced learning both on the manager’s part and on the team’s part.
Even when priorities would change, he’d let us complete our current work where possible, so that we wouldn’t have a mess of incomplete work lying around to try to come back to later. I’ve been on teams since where the team has been repeatedly interrupted to the point of not really delivering anything for months.
He assigned work to the team, and not to individuals, so the team would have to own it collectively. This kept the manager from being the choke-point for “what should I work on next?” questions.
He didn’t make decisions or solve problems for the team that the team was able to handle themselves (even when individuals on the team would ask).
He’d bring us the requirements, but leave the solutioning up to us. The result was that the team was largely autonomous and in control of how it accomplished its goals, so we immediately had a newfound sense of responsibility for our work and even pride in it.
He’d get estimates from us, and not the reverse (ie. deadlines). This way he could inform the business about what the estimated cost and schedule for a given feature was so they could make informed decisions about what features to pursue, rather than just delivering what was finished when the clock ran out.
He’d actively look for scenarios where the team was not “set up for success” and rectify them. Nothing is worse for team morale than a death march project, even a mini-one.
He held the team responsible for the performance of individuals. If someone on the team was struggling on a task (daily stand-up meetings make this pretty obvious) without any help from team, he’d ask the team what they were doing to help.
All of these things created the perfect climate for a high-performing team, but no one person alone (even a great manager) can create a high-performing team on their own.

The Team Member

The individuals on the team were also really impressive to me and no doubt absolutely essential to making the team what it was. Here’s a loose laundry-list of the qualities I witnessed and tried to adhere to myself:

Members of the team were open, honest, and respectful. They would give each other feedback respectfully and truthfully and would never talk behind each others’ backs. On other teams that I’ve worked on where this wasn’t the practice, team-crushing sub-cliques would form.
Once a decision has been made by the team, every member of the team would support it (even if he/she was originally opposed) until new evidence is presented. This actually worked well because we would always first take the time to actually listen to and consider everyone’s opinions and ideas.
Members of the team were results-oriented. “Effort” was never involved in our measure of progress. Software actually delivered according to the team’s standards was our only measure of progress.
The team would continuously evaluate itself and its practices and try to continuously improve them. We’d have an hour-long meeting every week to talk about how we were doing, and how we could improve (and how we were doing implementing the previous week’s improvements).
The team would practice collective code ownership, and collective responsibility over all the team’s responsibilities. This is actually a pretty huge deal; if people working on a top priority needed help, they’d ask for it immediately and anyone that could help would. If someone was sick, someone else would take on their current responsibilities automatically. If a build was broken, everyone would drop everything until there was a clear plan to fix it. The morning stand-up meeting would happen at exactly 9 am regardless of whether the manager was there.
The team would foster cross-functionality as a means of achieving collective code ownership and collective responsibility. What that meant was that members would actively try to teach each other about any particular unique skill-sets they had so that they would be redundant and team members would be better able to take on any task in the work queue.
The team would be as collaborative as possible. We all sat in a bullpen-like environment where we’d torn down the dividers that were originally erected between us. We’d often pair-program, not only for skill-sharing and mentoring, but also for normal coding tasks, because the boring tasks are less mind-numbing that way and the complicated tasks get better solutions that way.
In the 5 or 6 teams in our organization at the time, I think our team was by far the fastest, and most capable of handling the more technically diverse and challenging work. No one on the team was a rock-star programmer, but somehow despite that, the team as a whole was really outstanding.

Eventually when this manager moved on, I took the management role and ultimately failed to maintain that environment (partially for a number of reasons that were out of my control, and partially because at the time I didn’t realize the necessary ingredients). I’ve only really learned them since then by watching dysfunctional teams.

It’s a cliche, but in the right environment, the collective can indeed be greater than the sum of its parts.

EDIT (June 2nd, 2014):

I just moved my blog over from wordpress, but I don't want to lose the comments that I had there, because they're great and they're from the team that I was writing about. I'm just going to copy/paste them in here for posterity.

Todd said on May 8, 2013 at 8:12 am: Edit

You nailed it! I’m glad my simple ‘I miss working with you…’ email triggered you to write this thoughtful post. I admire your choice to spend time sharing your thoughts with the world.

I was trying to come up with some clever addition to your post but I keep typing and erasing. The one constant thought I whole heatedly believe had the most impact on our teams success is ‘Jaret’. I sure hope he doesn’t read this post, they’ll be no living with him! :)

Tom said on May 10, 2013 at 7:50 pm: Edit

Gregg and Todd, I also miss working with you guys. I do, however, have some pertinent hymns to add to the post. Though, I must first agree that a highly productive team is not formed from a cookie-cutter, but many ingredients do add to the potential success; kudos to Gregg for identifying most.

Management: I’d add that the “ideal” manager is an advocate for the team’s decisions, approach and methodology even if he does not fully understand them. Specifically, if the team can convince, through reason, that they were not smoking crack and that their solution would be suitable and economical (perhaps not in the short-run, but in the long-run), then he would go to bat for them. He’d do this no matter how much counter-pressure was exerted by upper management; clearly JW did this.

Secondly, a culture of experimentation, collaboration, and trust must trickle down from the upper echelons of management down to the lowest levels. Think of this as an approach to lowering entropy. Without this, all middle managers have their hands tied.

Team: Diversity in cultural and technological background is highly advantageous. The simple reason is that varied backgrounds allow differing opinions, analysis and approaches to a solution. I’d be bold enough to state that a homogeneous team is useless at being highly-productive because there is not enough diversity in the “knowledge gene pool” to forge great solutions. On the flip side, too much diversity may be a hindrance as agreement may be hard to come by for the final solution, but somebody should play devils advocate during the initial discussions; this only comes from varied experiences.

As well, team members should feel as if they are creating something of value. Perhaps this is more of a company or management issue, but I believe if people, regardless of their position in a company, feel like they are contributing to a greater good will naturally help their teammates and this will improve the productivity of the team as well as the firm’s bottom line.

I’m sure there are more comments to add, but I need to go have another glass of wine and reminisce :)

Gregg Caines said on May 11, 2013 at 8:12 am: Edit

Ha Good to hear from you Sykora :) I agree with your points. I think JW was set up for success when the “upper echelons” were supportive, and we were all destined for failure once those winds changed and upper management suddenly cared very deeply about the team’s every move on a day-to-day basis. Something makes me think that even if they had a reasonable ratio of right/wrong in their technical decisions, we’d still be irritated to be completely out of control of our own destinies, especially given our history of self-organization. Once the team gets a taste of autonomy, it’s hard for them to go back to command-and-control management. I suspect you of all people would agree. ;)

I agree with you on diversity too. We had a pretty diverse team but I think it worked well because we were respectful and trusted each other. I’ve seen teams since that were much less diverse and would argue disrespectfully about tiny things compared to some of the things that we would (calmly) discuss and (quickly) resolve. I’ll tell you one thing that I know for certain now: egos don’t make good team-mates. ;)

I agree with the “creating something of value” point as well, but I’m less about the value to the customer and more about the engineering craftsmanship. It’s the only way I’ve found myself able to reconcile product decisions I can’t agree with. Maybe one day I’ll have my greasy hands in the product side as well and get a chance to care all the way through though. ;)

tibi said on May 11, 2013 at 9:11 am: Edit

Well that was a nice post. And now I miss the team even more. I think you hit it right on the head (I can’t correctly remember the nail euphemism :-) )…

I’ve been looking for the same dynamics in a team ever since, and I think I’m getting close, but there is always something missing. Perhaps it’s the Tim Horton’s cup arch…

Jeff said on May 11, 2013 at 3:53 pm: Edit

Some great thoughts here Gregg – on the money pretty much across the board. A team is definitely only as good as it’s leadership allows. I’ve been in situations where the core team was incredible and worked together well but ultimately failed for the various reasons you any Tom both mention.

p.s. Hiya boys – hope everyone’s doing well. Did someone say Tim’s?

Gregg Caines said on May 13, 2013 at 9:01 am: Edit

Tibz and Jonesy! Good to see you two are still alive. :)

Chris said on May 14, 2013 at 7:06 am: Edit

It was something special to watch and interact with.

You do not really know what you’ve got until it’s gone, eh? There was a sense of what was happening but probably also a sense that it would fairly easily be recreated.

My current co-workers are tired of hearing how “we would have did it at my last place.” ;) Somehow it is not fair to hold other teams to that yardstick.

JW said on June 10, 2013 at 8:56 am: Edit

So many great things have been said above – and I have to agree with all of it. From my perspective, it really was a dream team. The trust, collaboration and commitment levels in the entire office were exceptional.

I have read numerous books on leadership and when I reflect on that team, we had all the characteristics of a high performing team.

For me, I loved the level of achievement, craftsmanship, and fun that we had. It wasn’t work – even a few of those late nights weren’t too bad because we were doing it together.

Is there a secret sauce to recreate that environment? Not that I have found. However, I have been lucky enough to be part of 4 different high performing teams over my career, and I believe there are things we can all do to help create an environment that could support a high performing team. From there – it is up to the team to choose to be high performing.

I identify well with the Patrick Lencioni’s book “The 5 Dysfunctions of Team”. Although I will not rewrite each of the characteristics from the book, I feel that we need to provide special mention to trust. It is the foundation of a high performing team. Nothing can be built without it. That means we need to know our team mates and their goals. How can we help each other? The more we help and recognize each other, the quicker the team will want to collaborate and achieve greater things.

We are all lucky to have been part of something great. I learned a ton from everyone there and I share those insights in my classes everyday.

Gregg – many thanks for starting this thread.

So I wrote a JSON API Framework and the Framework was the Least Interesting Part

2013-04-30T04:45:09Z

First a little background

I’ve always been a big fan of REST the way Roy intended it for two major reasons:

It uses HTTP the way HTTP was intended, playing on its strengths and not wasting any time re-inventing what HTTP already has. The state of HTTP APIs these days is really ridiculously bad, given what we should have learned from the unfathomable success of the WWW. I was also envious as hell at the great framework Erlang developers had in Web Machine . If you write HTTP APIs for a living and you haven’t seen HTTP mapped to a state machine, you really owe it to yourself to check it out. It’s brilliant.

The thing I really wanted to do though was make web APIs not so terrible. There are a few reasons that I think they’re terrible:

You have to read reams of documentation to do the simplest things, like finding the URL for something.
You have to hard-code urls all over your client app.
When you do something wrong, there’s no consistency in error output.
Knowing HTTP doesn’t make it any easier to predict how something will work.

If you rely on someone else’s API in your day-to-day work, you probably know what I mean. I’m not going to name any names (but here’s a clue: one of the worst offenders starts with an F and rhymes with Facebook).

So the APIs I wanted to make and to use would be:

semantically consistent with plain old HTTP
discoverable, like other pages and controls on normal HTML web sites are discoverable.

Luckily HTTP is also the vanguard of API discoverability, most often in the form of hypertext (So that’s what that H stands for!) . I’m a bit blown away that this wasn’t obvious to me earlier, but luckily I stumbled into Roy Fielding talking about hypermedia and I realized that that’s precisely the correct way to solve the discoverability problem. He was saying real REST APIs should minimize out-of-band knowledge by leaning on what HTTP’s already got: links! [^1] .

Then I saw Jon Moore make it real with an XHTML API . XHTML forms are a method of discoverability as well?!? My mind was blown. Think about that: XHTML documents tell you what kind of write-possibilities the API has. You don’t need to read docs to find out!

And then I saw Mike Amundsen bring it all back to json and I knew it might be something I could actually use in the real world and people trying to Get Stuff Done wouldn’t want to hunt me down and murder me in my bed (especially since I’m also a card-carrying member of the People Who Want To Get Stuff Done Liberation Front). I think most of us are of the belief that XHTML isn’t going to unseat json as the preferred media-type for APIs anytime soon.

Armed with all this inspiration, I wrote a web-framework. I ended up picking node.js mostly because the HTTP line-protocol stuff was already covered there without any extra cruft to get in my way, so I was free to try to aim for “Web Machine with json with links”[^2], which I call Percolator. The framework isn’t the interesting part though; it’s the stuff I learned along the way, which can be done in any framework/language/platform, even if it’s not baked into the framework.

The First Interesting Part: Hypermedia apis are indeed awesome

I started by just putting links in my json to other endpoints that make sense. That made my json APIs so much easier to use, even when the user is me. I had one endpoint for my entire API and every other endpoint was discoverable by following links. I used JSONView for Chrome and for Firefox and I could click around my entire API testing the GET method on every endpoint just by clicking. Here’s how it looks:

Those purple links are visited links that you can actually click on and get a new API response. I showed some co-workers that were using my API and eventually they stopped asking “Hey what’s the URL for getting X again???” They could always easily re-find it themselves.

Adding links also pays off when humans aren’t involved. In many cases it means that the client doesn’t need to construct its own links. For example: when one of my APIs was being used for replication by a client app the client developer asked how we’d handle pagination, because he wanted to page through all my results to replicate my API’s data elsewhere (sort of like how couchdb replication works, which is simple yet amazing ) . I had a pagination scheme working, but it didn’t make use of the database indices as well as I wanted so I told him instead to just follow the ‘next’ link and not to worry about the parameters for constructing the URL at all and that the absence of a ‘next’ link would indicate the end of the results. What this meant for me is that when I changed the pagination scheme to something more friendly to my database, I didn’t even have to tell him. What it meant for him is that he never had to change anything on his side to support the new form of pagination, and he didn’t have to construct any urls.

It blows my mind that pushing this simple technology further could end up with clients ONLY needing to store the root URL of the API, and not a who’s-who of possible disparate endpoints that practically mirrors the routing table on the server. People are actually starting to push the envelope this way. Wouldn’t it be great if at some point we could stop writing client software that’s hopelessly service-dependent ? Wouldn’t it be great if one day client-side url construction was considered as silly as using regexes to get data from json?

I should also add that even if you’re writing both the client and the server, which happens a lot with single-page apps and mobile apps, you might be tempted to think this is a waste of time, but in my experience it makes testing and debugging much easier and helps me remember what my API does after a month without using it despite my terrible memory. That minor extra effort of adding links pays for itself over and over again.

Interesting Part #2: JsonSchema is surprisingly awesome

Of course once I could surf around my API, it didn’t take long for me to bemoan the fact that I could only peruse GETs this way. I had seen Jon Moore’s XHTML API demo (This is my third link to it here, so you know it’s worth a watch!) , and I was jealous of how XHTML forms could describe the POSTs as well. Originally I decided that json needed forms as well, but the big brains in the Hypermedia-Web Google Group convinced me that I could just expand my link concept to know about other HTTP methods. And I had already stumbled upon JsonSchema as a way to specify what was expected in a PUT or POST payload, just like XHTML forms can do (but still using json of course).

JsonSchema is a particularly elegant solution to the problem, because I can write all my validations in JsonSchema, publish the validations as json to the api (JsonSchema is just json afterall), and then on the client side re-use them for validations before I even bother with an http request to the api. This way I can use the same validations on the client and server, AND communicate them naturally as part of the json payload. The end result was that my json APIs could now fully communicate everything that was possible via the json API itself. Here’s how it looks:

I added baked-in JsonSchema support to Percolator right away of course, but you can do it easily in any of the most popular languages ( see the validators tab). Currently I haven’t implemented any clients that use it for validations yet, but I do use it for my server-side validations. More importantly still, it communicates to humans everything that’s necessary for a successful PUT or POST. [^3]

Ultimately the addition of JsonSchema lead me to add a static frontend to Percolator that, after abstracting it away, I published separately as the Hyper+JSON Browser . You don’t need it, but you can use it in conjunction with Percolator apps to ‘browse’ them even more easily than JSONView does (and without installing a plugin). Any app (using any language/framework) that spits out the same sort of output as Percolator could use this the Hyper+Json Browser just the same, because it’s a static single-page app that works over any API that works similarly. Here’s how it looks in use:

When you click on that ‘create’ link to POST, it does this:

Browser after clicking the POST link

That red json logo indicates that you haven’t entered valid json (yet). Pressing OK will actually POST the json that you enter and show you the result.

Here’s a view of another resource (an individual item in the list from the previous collection resource, which can be found by clicking that item’s 'self' link. It shows delete and update links for this particular resource.

People have likened this to the Django admin interface, which is a huge compliment, not only because the Django admin interface is pretty legendary, but also because this frontend doesn’t know a single thing about the API other than the requirement that the API uses the aforementioned link format. This is an area where I’m really excited: In what other ways can other programs use these APIs without foreknowledge? (I’m not really talking about intelligent agents – they’d need to understand a little more than the API provides — but dumber things that still use human interaction eventually, like automatic HTML form generation, are mostly possible.)

(Somewhat less) Interesting Part #3: Object-oriented routing leads to less stupid stuff

I hate to sound old-school, but sometimes the object-oriented paradigm maps to a problem better than a functional one. In this case, I’m talking about routing. I don’t know why the HTTP methods GET, POST, PUT, etc are named “methods” but it sure does make perfect sense, because each one is tied to a particular “resource”: that “thing” you’re interacting with by using the HTTP methods for a given URL. It should be obvious then that HTTP resources should be the thing we route to, and not disparate methods. REST APIs are supposed to be the antithesis of RPC, but when routing involves the method as well, developers tend to make less RESTful APIs and framework developers tend to mess up some of the finer points of HTTP. Here’s a quick test:

Grab your favorite web framework and create a resource with a single GET handler.
POST to it. Did you get the proper 405 response?
send an OPTIONS request. Did it tell you that only GET was available?

Probably not. Web-machine gets this right because it routes to the resource, instead of disparate functions that when imagined together form a resource. I wrote Percolator to work like web machine in this way, because I wanted my APIs to look like something other than a haphazard collection of RPC calls without requiring a lot of self-discipline. The result is that Percolator responds with 405s when a client tries a method that that resource doesn’t support, and also responds to OPTIONS requests with a list of all the allowed methods. The result again is better discoverability without reams of documentation, and better feedback for client developers when they’ve done something the API doesn’t support.

There are advantages to OO routes for application developers to though, organization-wise:

All routes for a particular resource are connected to that resource, so there’s no chance of sloppy intermingling in the source code.
A bunch of code that you have to write in each method can be written just once for the resource.
You start to see patterns in what resources do, which can lead to certain classes of resources that can be re-used over and over, sometimes wholesale and sometimes with particular strategies that change them slightly. This is a bit of a big deal, in my opinion because it’s extremely rare that I see this kind of re-use in web applications in the real world.

Anyway: Percolator is indeed ready for production use (at least for the use-cases for which I’ve used it in production!). It’s extremely fast and extensible, and I hate it a lot less than anything else I’ve used for json REST APIs.

Hopefully it helps someone make a nice API, or it gives some other framework developer some new ideas. The APIs we have today cannot possibly be the best that we can do.

[^1]: plus a pile of other conventions

[^2]: Ultimately I diverged quite a bit from Web Machine, but it was always a huge inspiration.

[^3]: Alright not everything, but a heck of a lot. Linked documentation on my RELs would be another huge boost to discoverability.

3 Terrible Anti-patterns for Error-Handling in 'REST' APIs:

2013-04-22T05:22:07Z

I’ve seen these anti-patterns in a lot of web APIs, so I thought I’d detail why they’re bad and what to do instead.

Using a 200 status code for every response:

This anti-pattern is common for SOAP-like RPC APIs. Using a 200 status code for all your responses is silly because you end up having to reinvent an error-indication in your response body instead, and nobody in the world knows your new standard (or wants to have to learn it).

It’s also extremely confusing in the cases that an error has occurred because a status code of 200 means everything is A-OK. There are already a bunch of error codes in the 400 and 500 ranges that are fully capable of indicating all sorts of common problems. It’s still great to put an error message in the response body, but there should be a 400 or 500-level status code so people know that an error even occurred and don’t go ahead and try to interpret the response body as normal or otherwise assume a success.

Using a 500 status code for all errors:

This anti-pattern is common because 500 is what most frameworks will do when a web application throws an uncaught exception, and so it becomes convenient to just throw exceptions and not worry about the repercussions. This is mildly better than using a 200 status code for literally everything, because at least it shows an error occurred but it still causes a lot of confusion for client developers.

When you 500 for an error (even if your web framework did it for you), you’re signalling that the problem with the request was the fault of your application (the server) and it’s possible that retrying the request will be successful. Because 500 errors indicate an internal server error, they shouldn’t actually ever happen though[^1] — In general, you’ve got a bug in your application if they do, and the web server will tell everyone about it. The right thing to do, if there’s something wrong with the request is to respond with the appropriate 400-level status code.

Using a 500 error when a 400-level status code is more appropriate is obviously really confusing to the person making the requests because it doesn’t even let them know that their request needs to be fixed. Even if that person is also you, it forces you to start digging into your back-end for what could possibly just be a bad client request. If you don’t know which 400-level code to use, at the very least use 400 (Bad Request) itself. This will at least give your client developer an indication if the blame is on the server or the client and cut your debugging in half. A few words about the error in the response body can go a long way as well.

Logging 400-level codes at ‘error’ level:

Logging is pretty important to measuring the quality of implementation of an API, and learning about specific defects. The ‘error’ level in your logs should be a way to indicate “things that should not have happened that are within this application’s control”, so that you actually have a hope of attempting a 0-error policy. 400 level codes are errors of course, but they’re client errors, and not your server application’s error, so they should not be logged as errors along with your actual 500-level errors (Of course you might still log them as “info” level to keep track of what kinds of crazy things your clients are doing).

Why do all these details matter?

When you have an interface between two applications (the client and the server in this case) like HTTP, it REALLY simplifies things a lot if they communicate whether messages are acceptable or not (and why). Even if you control both ends of the interaction, you can save yourself a lot of server-side debugging by having the server indicate what exactly went wrong with a given request, instead of repeatedly reading stack traces every time some JSON you POSTed is missing a property. Compared to how little extra work there is involved in communicating the problem correctly, it’s pretty short-sighted to just let all exceptions bubble-up, or catch all exceptions silently (and 200).

Keep in mind too that if you’re now convinced that communicating errors is important, then HTTP already has a system for just that: HTTP status codes (plus whatever specifics you want to additionally communicate in the response body). Status codes are part of the greater HTTP standard that a lot of developers, libraries, proxies, and caches already understand. Re-inventing your own error mechanism is generally a waste of everyone’s time (with you at the top of the list).

Getting this right is a real time-saver over the long-haul even if you’re the only consumer of the API writing both client-side and server-side. The time-savings only multiplies as you add more clients.

[^1]: Sometimes 500 is appropriate of course: If your application requires a database and its lost connectivity to it, then a 500 makes total sense. It signals that there’s a problem in your application and the client can possibly try the request again later.

The Node.js Community is Quietly Changing the Face of Open Source

2013-04-13T13:46:06Z

If you consider the number of packages on pypi, npm, and rubygems, and the release dates of python (1991), ruby (1995) and node.js (2009), it looks a lot like those communities are getting new packages at the following rates, per year:

python: 29,720 packages / 22 years = 1351 packages per year

ruby: 54,385 packages / 18 years = 3022 packages per year

node.js 26,966 packages / 4 years = 6742 packages per year

It’s important to note that I’m only showing the other languages as a measuring stick. There are probably a lot of reasons for the disparity here (including my imprecise math), and I’m not trying to say anything like “node.js > ruby > python” but obviously new node.js packages are being published to npm at a staggering rate. I just checked npm (on a Sunday night) and 20 new versions of packages have been published to npm in the last hour alone.

This is a pretty amazing feat for node.js. How is this possible?

Batteries NOT included

If you’ve needed an HTTP client in Python, you’ve probably asked yourself “should I use urllib, urllib2, or httplib for this?” like so many people before you. Well the answer is probably that you should use requests. It’s just a really simple, intuitive HTTP client library that wraps up a lot of weirdness and bugs from the standard libraries, but it’s not a standard library like the others.

The “Batteries included” philosophy of Python was definitely the right approach during the mid 90’s and one of the reasons that I loved Python so much; this was a time before modern package management, and before it was easy to find and install community-created libraries. Nowadays though I think it’s counter-productive. Developers in the community rarely want to bother trying to compete with the standard library, so people are less likely to try to write libraries that improve upon it.

Developers are less likely to use non-standard libraries in their projects too. I’ve heard many instance of people suffering through using an inferior standard library just to have “no dependencies”. But in this day and age of cheap massive storage and modern package management, there are very few reasons for a policy of “no dependencies”.

Conversely, the node.js core developers seem to actually want to minimize the standard library as much as possible. They have repeatedly removed features from the standard library to free them up to be implemented by the community instead. This allows for the most variety and lets the best implementation win (but only until someone makes a better one, of course!).

Imagine how liberating this is for standard library maintainers too. The node.js standard library is comparatively tiny, freeing the core developers to just deal with the core necessities.

The Tiny Module Aesthetic

Just like the 140 character limit on twitter makes people “blog” more, the node.js community has a culture of publishing tiny modules that makes people more comfortable with publishing smaller packages, and so it happens vastly more often.

While I don’t really consider myself “a node.js developer”, I’ve found myself publishing over a dozen packages in the last year myself, just because I’m working on a node.js project at work, and I publish any reusable package from that work that I create. My project ends up factored better, and I end up with a bunch of building blocks I can re-use elsewhere (and then I do!). Sometimes random people from the interwebs even fix my bugs before I notice them!

This approach is not unusual at all either, at least in the node.js community. The community is lead by developers that have each published literally hundreds of packages. I don’t even know how this is possible.

Of course there’s absolutely nothing to stop other languages from following suit, so it seems to me like community culture is the deciding factor.

Massive monolithic frameworks like Ruby on Rails tend to not take hold in the node.js community mindshare, I think due in part to this “tiny module aesthetic”, and it’s all the better for it. It seems to me that monolithic frameworks like Rails don’t have a clear notion of what does not belong in Rails, and so they tend to expand indefinitely to include everything an application could need. This leads to people trying to assert that Rails is the best solution for all problems, and having a cooling effect on innovation within the web application space in the Ruby community.

The tiny module aesthetic also leads to an ecosystem of extreme re-use. When a package doesn’t do more than is absolutely necessary, it’s very easy to understand and to integrate into other applications. Conversely, while massive monolithic frameworks tend to appear in a lot of end-user applications, they rarely never end up being dependencies in other packages themselves. It seems unfortunate to me that so much great software should be so difficult to re-use.

git and github.com monoculture

To be honest, few things are more boring to me in software development than version control, and I find the complexity of git to be a bit silly and mostly unnecessary for me, but I’ve never heard of a node.js developer that doesn’t use it, and there’s huge value in that kind of monoculture. It means that they all speak with a common language and there’s no impedance when you want to contribute to someone else’s projects.

Github.com has a similar effect in lowering the barriers for cross-pollination between developers. I’ve only very rarely seen a node.js project that wasn’t on github. This means I immediately know exactly where to find source code if I want to contribute.

The pluses and minuses of a github monoculture go far beyond this, but there’s probably enough fodder there for an entirely different post.

Permissive Licensing

Node.js packages tend to use extremely permissive licensing like MIT and BSD licensing. In fact, the default license for a package created by npm init is BSD. I think this is another sign of the times.

Very few people care anymore if someone else forks and doesn’t contribute back. There’s very little to gain in not contributing back anyway because of the cost of maintaining your own fork.

This is important because no one wants to have to deal with dependencies that make them think about legal repercussions, or give them extra responsibilities. Licenses like the GPL end up having a cooling effect on software-reuse for this exact reason.

EDIT Jun 3rd, 2014: Comments below are basically copy/pasted from my old blog format for posterity.

30 thoughts on “The node.js Community is Quietly Changing the Face of Open Source”

max ogden said on April 14, 2013 at 10:25 am:

Nice writeup! I do think that the term ‘node community’ is a little bit too general since some of the most popular node modules are in fact bigger than they should be.

For example, the 4 most popular modules from https://npmjs.org/: underscore, async, request and express could all be split up into smaller pieces:

Underscore isn’t terribly large but it has things tucked in it like a templating function that don’t belong.

Async is thematically consistent, offering many variations on a theme, but from the empirical usage patterns that I’ve seen most people use a small percentage of the offered functionality.

Request actually went through the modularization process over the last month or two and went from having 1 or 2 dependencies to having 11.

Express is perhaps the greediest module on NPM (IMHO) in that it invented its own naive APIs early on which means that now you can only use modules designed for express with express since the extensibility is stuck in a relatively pre-cambrian era. Classic framework lock-in, albeit to a lesser extent than other, larger web frameworks. I’ve thought a lot about why this happened and believe that if you try to make a land-grab early on and abstract too much too early then you get stuck with your abstractions due to backwards compatibility. Good software grows slowly.
Gregg Caines said on April 14, 2013 at 1:30 pm:

Yeah I totally agree, and you’ve probably explained it even better than I did.

Express is actually a bit of a conundrum for me for a few reasons, especially since in trying to create a mini API framework myself ( https://percolatorjs.com ), I’ve felt forced into duplicating some of the things I dislike about the modularity (or lack of modularity) in express, to the point that I eventually gave up and added ‘connect’ middleware support. There are a few competing pressures that make web frameworks like express (and rails for that matter) popular:

(1) Some people just want to “get shit done” ™ and don’t want to have to search for multiple pieces of a puzzle to hobble together a solution. I think there’s probably a gap where finding the best modules for a given problem is still a too hard. Filling that gap (somehow) will promote collaboration even more. The current npm search (on its own) doesn’t solve the problem IMO.

(2) HTTP is actually really complicated and very hard to abstract effectively. For example, how many frameworks 404 when you try an invalid method on an existing resource instead of the proper 405? This is mostly because of the way routing is abstracted; These frameworks use the http method as the first ‘key’ in the routing table, when really the url makes more sense as a key and the method should be secondary. There are a number of cross-cutting aspects like this including logging, error-handling, and authentication where some sort of centralized pre-processing makes a lot of sense. Not all “aspects” should be handled with serialized pre-processing middleware though, and unfortunately that’s what we’re seeing a proliferation of. I’m disappointed that I can’t build a mini-framework without integrating connect, or rewriting/repackaging all those otherwise great connect middleware modules out there, but I don’t know a better way. To be honest, I don’t even know at this point what the better abstraction would be. Doing everything in pre-processing middleware and monkey-patching the request/response objects doesn’t seem like the answer though.
Edwin Knuth said on April 15, 2013 at 10:41 pm:

Good stuff. I haven’t felt this excited about a platform community in a long time.
Ronan Berder said on April 15, 2013 at 11:37 pm:

Good read: spot on.

I would add a couple additional factors to that trend:

- Proper package management: `npm` was and still is what got me hooked up. Package management is often an afterthought that is rarely properly addressed.

- Flat community: which is partly illustrated with the extreme culture of re-use you mentioned. I feel there is much lower friction in creating and contributing to modules in the node.js community, with no gate keeper in sight.
Mark Smith said on April 16, 2013 at 12:26 am:

For a similar system: Perl’s CPAN (the package repository) has been online since 1995 and presently has 120,447 modules for a yearly rate of 6691 modules for 18 years now.

It’s been providing the same functionality you mention: an encouraged license (“the same terms as Perl”), a monoculture of software (CPAN for package distribution/documentation), and an emphasis on tiny modules. Little packages are encouraged and really the way to go when you’re building Perl modules.

That said, it’s great to see those practices being adopted successfully by Node and seeing the success of the groups using it. We live in very exciting times for open source. :)
Edwin Yip | dev of LIVEditor said on April 16, 2013 at 3:02 am:

The “The Tiny Module Aesthetic” reminds me that, the emerging of the “Single-purpose app” in recent years, and the design philology of the Unix utilities. And I think those are all about “simplicity”, which I think and care about the more and more in recent years, but I always have to remind myself about when writing code for LIVEditor – my code editor with “Firebug” embedded.
Mansur said on April 16, 2013 at 3:21 am:

There are some frameworks in Node.JS which are just copied, not unique, like for example https://github.com/biggora/caminte – is just a copy of Juggling DB https://github.com/1602/jugglingdb.
I would strongly recommend to use genuine frameworks which have been tested by the community and better supported.
Chankey Pathak said on April 16, 2013 at 4:11 am:

Why don’t you consider Perl? 120,460 Perl modules in 15 years, that makes it 8030 modules per year and it’s increasing exponentially now.
PeterisP said on April 16, 2013 at 4:16 am:

“Batteries not included” approach puts a significant cost on application developers.
If I need to do a simple thing X, I’d rather have a single good library for X rather than twenty libraries with varying focus and quality. Even if one of those twenty libraries would be far better for my needs, the task of properly comparing and choosing the lib is often higher than doing X with a random library.

It also reduces maintainability – if one project uses lib1 for X, and another project uses lib2 with a different API for the exact same thing, then it makes stuff harder to maintain.
Gregg Caines said on April 16, 2013 at 6:29 am:

@ Chankey Pathak, Mark Smith: For the record, I didn’t include perl because I haven’t got any perl experience to draw from, while I’ve spent years writing reams of code in ruby, python and node.js. I can honestly say though that CPAN is nothing short of legendary even outside the perl community, and npmjs.org would do well to follow its lead. It would be interesting to know too how CPAN maintains that rate of growth, and if it’s due to the same things I’ve mentioned.

I should also add that node.js isn’t even the language/platform that’s nearest and dearest to my heart. I’d really just love my other favorite languages’ ecosystems to pick up some of these qualities.
Su-Shee said on April 16, 2013 at 8:47 am:

From a CPAN’s lover’s perspective NPM already is _very_ well done and pretty smooth to use.

From a Perl lover’s perspective, JavaScript doesn’t have a choice anyways, because you really NEED a lot of modules to work around well let’s politely say “language quirks”. ;)

You simply need things like underscore or node, because it gives you things which aren’t builtin in the language. (*cough* npm install printf *cough* ;)

Similarily, Perl has gotten its Lisp-style MOP OO module Moose, because there’s no such thing as a “class” in Perl.

Our equivalent of underscore-alike modules are for example List::Utils, Lists::MoreUtils etc. etc. – and you can’t possibly shove ALL those modules into what we call “core” – set aside that half of the people want Foo and the other half Bar and then there’s our “problem” we’re really proud of: Backwards compatibility is _amazing_ in Perl and there’s always someone who runs a 12 year old Perl and still needs his Really::Ancient::Module::BlahBlah. ;)

So far I encountered exactly ONE npm module which I couldn’t install blindly with npm install -g – and that was due to an architecture check in its requirements (which isn’t necessary, module very high level) and my Linux flavor returns a different architecture string – so one lesson might be to encourage a best practise “don’t overdo requirements for your module, it just complicates things”

In Perl, it’s common that you simply ask “the community” (IRC for example) “what do I want for XY?” and get a variety of 2-5 modules and a short explanation why you want A and not B. Imagine for example you have more than one XML/HTML parser module (which we do) – do you want XPath support or do you prefer DOM-style methods? Do you need a strict parser or do you want to work on dirty HTML?

Most module authors mention these kinds of use cases and “flavors” to do things and refer graciously to the alternatives. Plus, there’s the entire commenting and rating system for modules and yes, you of course look if Perl gurus Tom, Dick and Harry Superperl ranks module Foo highly or not – it’s not that we don’t have our own little rockstars and divas. ;)

Due to variety and so many choices, there’s also bad choices and we put A LOT of work into discouraging certain modules these days which are simply not really appropriate to use in 2013 or some well known badly written ones. But you can’t just throw them out, too – because Perl is installed everywhere – what is the browser for JavaScript is pretty much any Linux/Unix for Perl. And all the shitty JavaScript quirks you’ve gotten from browser vendors, we get from distributors which come with a Perl from 1999 in 2013.

Sooner rather than later (there already are a couple) there will be a class of modules on NPM which are dual use – browser and Node-based server-side and you will have to keep that straight and compatible and that will be a lot of work. Also, what if people start writing JavaScript modules which are able to run in browser, on Node AND on e.g. spidermonkey? (what about Rhino/Nashorn?) I’d encourage that a lot, not everything needs Node and NPM is the de facto thing to put a module up on by now.

Another advantage of CPAN is: It’s _extremely_ centralized. You look for a module on CPAN. Period. There’s no “download from here” or “fetch from there” – it’s on CPAN or it doesn’t exist. As CPAN is tested and tested and retested by volunteers, using modules from CPAN pretty much ensures been tested on a dozen platforms. If you use the plain “cpan Foo::Bar” installer, you can always watch all tests live while running the installation of a module.

And Perl modules have tests LIKE THERE’S NO TOMORROW – as Perl’s testing flavor TAP is extremely easy and simply and cheaply to do, modules actually _do_ have a LOT of tests.

Last but not least: On average, the commonly used Perl modules are _extremely_ well documented and the docs can be read on commandline and via web. The documentation style is probably rather old-school from a Node lover’s perspective – it’s more on the dry, informative side and less about being cute and funny – but some docs literally go into the amount of a book written.

Also, no matter how big a project – I simply expect it to be installable via CPAN. There’s a couple of Perl things which are more applications but “modules” and are not installable via CPAN, but that’s pretty rare. Entire frameworks are installed via CPAN – think like the difference of how you install Derby.js and Meteor.js.

And: I run deliberately large dependency-chains for the fun of it – CPAN has a flag for “resolve all dependencies yourself and say auto-yes to everything you’re asked during installation” and I manage a couple of hundred modules in ONE session WITHOUT any error while running ALL tests during installation – possibly on half a dozen Perl versions (Made with perlbrew which is our nvm/rvm/virtualenv).

As the Perl mantra “there’s more than one way to do it” is done by the JavaScript community on the same level of excess (a dozen web frameworks anyone? ;) you’ve made your choice already anyways to go down the road of extreme flexibility and variety. It already happened, no going back here.

The Perl community for example is very forceful to really push people into using modules from CPAN exactly BECAUSE they’re tested and have solved hundreds of problems for you – you don’t split a CSV file by hand, you use Text::CSV which is proven to work for a decade. You won’t do better with your meager little split. :)

You don’t parse HTML with regex, you use a module. You don’t validate email addresses by hand, you use a module. We have modules like Regex::Common for example which contain a range of common usecases people usally throw (bad) regex on or modules which pretty much read any stupid format known to mankind – it’s all put into modules and more modules.

Luckily, Perl also has most of the necessary modules I usally call “boring shit” modules – Excel reading and writing anyone? Those are the true heroes in my opinion – people who sit down and really write this kind of ungrateful, boring stuff you suddenly need there’s *gasp* lurking an “enterprise” around the corner – not to mention that nobody in their right mind is going to implement SSL bindings three times. (One lesson for JavaScript/Node might be to maybe not add another web framework but deliberately go for the boring stuff sooner rather than later and get it over with..)

For my part, I can’t rave enough about CPAN and some of the people poured their hearts and time and knowledge into some modules. Billion euro/dollar businesses run on it – some for over a decade straight.

So, hack, upload and enjoy. :)

Yours truely – a Perl lover. :)

CPAN’s main website: http://metacpan.org
subdigit said on April 16, 2013 at 9:19 am:

Maven is also something that’s begun to pick up pace in the Java community. Stats seem to be here: http://search.maven.org/#stats

But not sure about exact growth rate. A pretty picture is available here: http://mvnrepository.com/

While having a small standard library is great for essentially encouraging reinventing the wheel to make it better, it can at times be a detriment for “just getting things done”. It’s called a “standard” for a reason, which is that you can rely on it to be present and available for use, and more importantly, for understanding by the developers that come in and out of a project.

That being said, popular extensions will solidify and essentially become a 3rd party standard for use and becomes indispensable and tied into the core project, whether it’s just the common-knowledge standard du jour, or it actually gets merged in.

It certainly is a more interesting way to get a diversity of development of what people consider to be core though. For that, I find that model interesting to watch and see how it grows and if anything I favor “wins” :).
Gregg Caines said on April 16, 2013 at 9:37 am:

Su-Shee: thanks for the education! I agree, CPAN sounds great. Since we’re getting along so well, I won’t say anything about the language itself. :)

A few minor clarifications on javascript in 2013: Don’t `npm install printf`, use util.format: https://nodejs.org/api/util.html#util_util_format_format . Underscore is much less necessary nowadays (especially in node.js) as a result of ecmascript 1.6 improvements (See “iteration methods” at https://developer.mozilla.org/en-US/docs/JavaScript/Reference/Global_Objects/Array ). Also using jslint will keep you 99% quirk-free (I have it integrated into my builds and vim).
Rudolf Olah said on April 16, 2013 at 10:30 am:

There’s no need to slander the GPL in this post, it’s just as permissive and would allow the same type of collaboration. The trouble is that most developers also want to use libraries as part of proprietary code bases. I don’t understand why they don’t just dual-license; GPL for non-profits and devs, MIT/BSD for commercial/proprietary software.

There really should be a bigger push to license internal code under the GPL. 99% of the time it doesn’t offer a competitive edge. It’s the data and marketing and network effects that have more to do with it than anything.
Andrew de Andrade said on April 16, 2013 at 10:46 am:

Great post. I’d also add to that list the benefits of local by default packages that are in your project folder. The approach of tools like easy_install, pip, bundler, etc is to hide all your depemdencies in some dot-prefixed folder outside your project directory. This promotes an out of sight, out of mind mentality. The npm approach on the other hand puts them in your project and makes it much more likely that you are going to explore the source of your dependencies, more likely that you bug fix those dependencies instead of work around bugs and most importantly you’ll treat that folder like part of your own project. This last point means that you are likely to develop one of your own generic modules to the point where it’s good enough to submit back to npm as a public module.
Su-Shee said on April 16, 2013 at 11:03 am:

Oh, go ahead and say something about the language and I will probably tell you the exact same thing back about JavaScript – or something quite similar. If anything, both have A LOT of history and baggage – but they both also have A LOT of flexibility, otherwise the range of modules existing as of today wouldn’t even be possible and people wouldn’t deliver at that speed. And let’s face it: We can happily unite (Come along, you too Java, don’t be shy…) in Python’s and Ruby’s dislike for {} and ; – and we use a lot of them. ;)

Sadly, format.util supported too few options when I looked last. (In the end, I did it in DB anyways..) – but it’ll evolve, I’m sure.

Underscore will probably move to other list-supporting functions over the years I would expect, the map-grep-filter-foreach range is just the basics anyways. (At least, Perl didn’t need map-grep-foreach-shift-unshift-push-pop-splice as a module, that really is core-core…)

And before someone is tempted to bitch about Java-and-Maven: _THAT_ community delivered literally EVERYTHING I can think up of “boring shit” and “enterprisey stuff” you can imagine. And if you ever looked for “good PDF manipulation” for example: Java did deliver. Kudos for that.

If anything the lesson simply is: Yes, you want TONS of modules, yes, the environment IS important, no, it doesn’t matter if you’re a less-than-perfect language, as long as you support a certain flexibility, people will work around it.

And style evolves a lot – nobody writes JavaScript like in 2000 anymore and you get discouraged a lot to do so – same goes for Perl. But if you already had a web business in 2000, you probably just can’t throw out your entire carefully crafted JavaScript or your backend of the day (in 2000) and that is something EVERYBODY faces. Perl did and we paid dearly, Java does, Ruby will, Python will do too.

JavaScript showed – thanks to Douglas Crockford and jQuery among others and yes, thanks to Node.js, too – how to evolve nicely into a new style – but also Node.js setups will at some point be really huge and not that easily updated anymore and then you face the very same thing everybody else does.

And god do I wish this petty bickering among languages would stop – we owe each other so many things – all of us – and all of us owe Lisp and Smalltalk and C.

Let me all remind you that there was a time when “real programmers” laughed about using a “scripting language” on this cheap little Linux hack with this meh little HTML thingie to put up a business on and now look where we all are today…
Gregg Caines said on :

Su-shee: Totally agree. Well said.
Gregg Caines said on April 16, 2013 at 12:39 pm:

Rudolph: “Slander” is a legal term and a pretty ridiculous one for what I said. More legalese isn’t going to help the GPL’s case. You already agree that the GPL is more restrictive for people mixing software with other proprietary software, so how can it *possibly* be slanderous for me to say it’s not as permissive?
Hisham said on April 16, 2013 at 12:49 pm:

Well, counting the age of the Python and Ruby repos by the age of the language (and not of the package management projects, such as pypi and rubygems) while considering node.js by the age of the project and not the language (JavaScript) skews the numbers quite a bit!

Let’s compare by language age:

python: 29,720 packages / 22 years (1991) = 1351 packages per year

ruby: 54,385 packages / 18 years (1995) = 3022 packages per year

node.js 26,966 packages / 18 years (1995) = 1498 packages per year

And by package manager age:

python: 29,720 packages / 11 years (2002?) = 2701 packages per year

ruby: 54,385 packages / 10 years (2003) = 5438 packages per year

node.js 26,966 packages / 4 years (2009) = 6742 packages per year

Except that, of course, these numbers are meaningless, because each language/tools has its own submission evaluation process (including “no evaluation”), and also different granularities in how to count “packages” (by module? by a set of modules that compose a project?).

Also, language-specific package managers are more or less a new thing — counting for example the number of packages submitted last year would be a more meaningful metric (apart from the problems I raised above.)

Still, it’s nice to see that language-specifc package managers are thriving, and I wish all the best to all of them. :)
TJ said on April 16, 2013 at 5:40 pm:

The reason Express is the size it is, is because all these things are what we would be duplicating each time anyway. Yes a lot of it could/should/will live in npm, but an aggregate is not harmful. Look at npm-www for example, this is what happens when you re-create things every single time you write an app, it becomes are large unwieldy mess of stitching together random things and re-inventing within the app itself instead of an aggregate “framework”. Most middleware should absolutely be abstracted to a lower level and then “wrapped” as middleware, there’s no question about that, but should they cease to exist? absolute not.
Gregg Caines said on April 16, 2013 at 8:07 pm:

TJ: Yeah, I actually agree that the core http module isn’t a useful enough abstraction to be used for many (most?) real world web applications. It also doesn’t offer a reasonable way to add functionality other than wrapping it or monkey-patching the request/response objects. Cross-cutting concerns are the obvious problem (security, logging, routing, etc), but also: what if you want to just add functionality that’s more easily accessible than constantly requiring the same 20 modules that are all very HTTP relevant? And I agree that larger aggregates are useful and valid (and I bet you’d also agree conversely that putting an orm in a web server abstraction is a superfluous aggregation, and so there are limits to what aggregates make sense).

I don’t actually have a better solution than what express does. I hope that was obvious from my comments. I certainly tried, but ended up coming back to connect. At the same time, I find it unfortunate that connect middleware have non-obvious interdependencies and are difficult to keep clash-free (basically all the problems of monkey-patching). But like I say, I haven’t got a better proposal.

I personally don’t think node.js has the HTTP abstraction problem solved very well (yet).

> Most middleware should absolutely be abstracted to a lower level and then “wrapped” as middleware, there’s no question about that

Not a bad start!
Chankey Pathak said on April 17, 2013 at 12:12 am:

@Su-Shee: If there would be an upvote button on comments here I would’ve hit it so hard that it would break my mouse! Well written, +1 to you.
Geoff Flarity said on April 21, 2013 at 8:38 pm:

Some times I get confused when people try to convince other’s programmers that easily re-usuable modules are awesome and you should try them. Then I remember that not every was into perl 10 years a go… Even if you’re not a fan of the perl syntax (beauty), CPAN is the shit and so is NPM.
Pingback: This Week's Links: _why Edition - NYTimes.com
Adam said on May 15, 2013 at 8:17 pm:

I’m sure this was already said.. but you can’t compare node.js to php/ruby. I am a node developer.. but node is a new FRAMEWORK. It would be more accurate to compare JAVASCRIPT to php/ruby.

Just FYI
May 15, 2013 at 11:17 pm:

Adam: The comparisons I make are comparisons in the customs of the communities surrounding those technologies and not the technologies themselves. I think at that level, it’s apples-to-apples.
Juzer Ali said on June 30, 2013 at 12:30 pm:

@PeterisP:
> “Batteries not included” approach puts a significant cost on application developers.
I agree, it sure wastes a hell lot of time.

@TJ: Is express code too succinct for what it does? For example, its very difficult to figure out what is happening inside a router https://github.com/visionmedia/express/blob/master/lib/router/index.js#L46
Pingback: To node or not to node – overview of Node.js | Developing Scalable Applications
Simon Boudrias said on September 13, 2013 at 10:17 am:

@Adam I have to disagree here. JavaScript is only a language (it is syntax), PHP/Ruby are more than syntax, it is syntax + all about the runtime and functions hooks to the environment. Node.js is JavaScript + runtime/hooks stuff. That’s why Node.js is comparable to PHP and Ruby, JavaScript isn’t.
Pingback: ~ The Truth about JavaScript. « Clint Nash

Release-ophobia

2013-03-16T14:33:09Z

If you’ve developed SaaS in a large organization (or even a small organization that acts like one), you’ve probably seen some of the symptoms of fear of releasing software:

Off-hours deployments and weekend deployments
Infrequent releases
extended code-freeze and testing cycles
Reluctance to refactor
Feature branches for the purpose of ‘cherry-picking’
release managers
many non-customer-facing environments before production.

These symptoms usually happen only in larger/older organizations, because of two factors:

They take a lot more personnel / time to implement than smaller organizations have.
They are usually processes and attitudes that are collected as a reaction to problems that occur in the course of a company’s lifetime.

#“But you can’t be too careful!” One problem is that you CAN indeed be too careful. It’s absolutely imperative that the costs be considered as well, if you want software development to still go reasonably fast. Otherwise these manual preventative processes will slowly creep in over time and your team will be releasing at a snail’s pace.

The other problem is that these policies alone are in some ways not careful enough. Where are the techniques for mitigating defects that inevitably still slip through? Where are the techniques to detect them, limit the customers exposed, and limit the time that customers are exposed?

All of these symptoms have a few things in common: They tend toward heavy manual and preventative process added generously over time, usually with little thought to the cost/benefit ratio of such process. It’s all too common for every production issue to result in a new process to prevent it, without regard to how infrequently that problem might occur, how damaging the result is, or how easily it is rectified.

So how do we keep bugs from getting to our customers?!?

Non-preventative measures (I call them “Reactive Measures“) do indeed exist, and can help you safely maintain a fast and steady flow of new features to customers.

Feature-Flags allow you to commit incomplete work to master and push it to production safely. Customer-specific feature flags allow you to test in production or enroll customers in a beta program. With a good feature-flag system, it’s even possible to release a feature to only a certain percentage of customers, and to watch your metrics to see the effects.
Easy/fast (read “Automated”) deployment. If deployment takes a long time, then you can’t afford to deploy often. Also, if you can’t deploy quickly, you can’t redeploy quickly (in the event of a problem).
Easy-undo. Rolling back from a defective release to a known-good release is the simplest and fastest way to recover from defects, if the process is both quick and easy. Defects will happen no matter how much prevention you use: Easy-undo can mean the difference between light customer impact and heavy customer impact.
Smaller releases. Smaller releases have less code, and therefore generally less bugs. Because they don’t contain many changes, they’re easier to test and a rollback is less-likely to negatively impact a customer. Also when a production defect appears in the latest release you have much less code to sift through to determine the cause.
Practice Continuous Integration (which necessarily involves always committing to master, and only incidentally has to do with using a CI server) . Feature branches, infrequent commits, and committing non-production-ready work are all practices that need to be avoided, because they are all time-consuming delaying mechanisms.
“Done” == “In Production”. If the team moves on to new features before the last one is in production, then that last feature is obviously unnecessarily being held back from release. They will be faster if they deliver to production and ensure it functions there properly before moving on to new features because it avoids unnecessary and time-consuming multi-tasking. This goes hand-in-hand with smaller, more frequent releases.
Comprehensive Automated testing. When you deploy more often, you need to test more often. The cheapest way to do this is with comprehensive automated tests. While they’re a heavy-upfront cost, if you run them as often as possible, they’ll pay for themselves very shortly.
Continuous Production testing. Create automated black-box integration tests for some of your most important features and figure out a way to run them repeatedly and continuously against your production deployment so that you know instantly when those features are broken. There’s no need to wait for customer feedback to learn about a faulty release.
Searchable, aggregated logging with proper log-levels. Making your logs easily searchable in one place makes them something that developers will actually use for defect detection and for debugging. They can go a long way to making your application more transparent.
Runtime monitoring, runtime metrics and information radiators (most importantly alerts). Instrument the things you care about most, and have alerts sent out when the measurements are abnormal.

Note: Of course some preventative processes are necessary, but when they are, automation is almost always preferable to time-consuming manual human processes. Either way, you should consider the cost of the process and determine if it the fall-out from the type of defect it prevents is actually worth that cost.

In the end, a fear of releasing is a fear of change, and a fear of change is a fear of improvement. This is a crucial lesson that Big Company software teams can learn from the small, scrappy start-ups that are quickly trying to disrupt them. The more often you release, the more often you can test hypotheses and get customer feedback, and the more often you can course-correct. Companies that don’t realize this are susceptible to being leap-frogged by competitors that do.

Your Team Probably Doesn't have the Same DVCS Requirements as Linus

2012-01-03T14:43:58Z

Version control branches come with costs. I’m at the point now that I don’t think it’s possible to get away without spending a couple of days every month managing issues with version control and merging if you’re using feature branches liberally as is common with DVCS (like git) users. I’ve spent days myself in the past months, so I felt compelled to write this because those were days I could’ve been actually doing something that adds value instead.

The purpose of branching is to delay integration of some code into the main code line (for various reasons). There’s a heavy cost that comes from this in that the less often you merge, the harder it is to merge. I’m blown away by the effectiveness of automatic conflict resolution in git, but the bottom line is that if two developers are working in the same area of the codebase, semantic conflicts will emerge that need the intervention of a human to resolve. These can be complicated and tedious and involve multiple developers. Yes, branching is easier with DVCS, but this actually exacerbates a fundamental merging problem that DVCS doesn’t solve.

“What if I keep my branches short-lived?”

There are some better defenses for branching, like “What if I keep my branches very short-lived?”, and while it does mitigate most of the problems with merging, it also mitigates any advantage you think you might get from branching. If you’re only going to branch for a day, what’s the point?

“Then how can I pick and choose what gets added/released?”

Merging early and often leads to simpler merges, but what about the case when you get partway through a feature and want to scrap it because you’ve come up with a better way? There are a number of ways to mitigate this problem, but if your need to “unmerge” occurs more often than your need to merge, you’ve probably got larger organizational issues to deal with. If those issues are purely technical, try more prototyping to uncover those issues earlier, and feature-toggles for better control over what gets released. If those organizational issues are non-technical, like frequent changing requirements mid-feature-development then you simply need to get better (and probably smaller) user stories before commencing.

“What if I do daily merges from the mainline into my feature branch?”

You’d think it shouldn’t matter which way you merge if you merge often, but that’s only fine if you’re the only one branching. As soon as someone else on the team comes along and starts up their own feature branch and follows your same best practice, the two of you are diverging from each other and creating merge-debt.

Using a continuous integration server is NOT sufficient for actually practicing continuous integration.

I’ve heard of some teams using a bot to auto-add all branches to the team’s CI server, making sure that each branch gets put through the proper rigor of the CI server. This is actually cargo-cult continuous integration though because those teams are not actually continually integrating at all. Merging early and often is a prerequisite of continuous integration, by definition. Instead, running multiple branches in CI supports maintaining those multiple distinct branches for even longer, so you’re actually using the CI server to undermine the ideals of CI. Even if you run a special build that tries to auto-merge all branches (yeah right!), but is not the mainline and is not intended for release, you lose the ability to regularly deliver work to some staging environment for manual regression testing.

A small non-distributed team doesn’t have the same problems as Linus

If your team is in the same room and practices collective code ownership (instead of having a central maintainer) you should just do in-person peer code reviews (or better yet, pair programming) instead of relying on pull requests. It’s wasteful to force every aspect of intra-team communication to go through the computers in the room. I would even go so far as to say that if you’re in the same room, you don’t actually need a DVCS.

Most of the reasons behind feature branches seem to be a cure that’s worse than the disease itself. It’s vastly simpler for us to commit to the mainline at all times, early and often, and deal with deactivating or removing particular pieces of code in other ways (like feature-toggles) as those problems arise (which does not in my experience tend to be very often compared to how often I am merging new stuff in that stays in).

I don’t mean to be negative about DVCS either. I actually use git exclusively now, I think github has been a huge boon to open source, and I love anything that makes Linus more effective at maintaining my favourite OS kernel. It just really seems like feature-branches have never solved any real problem that I have, while they have caused a lot of pain.

Agile: The Good Parts

2011-09-14T14:51:37Z

Let’s just forget about story points, pigs, chickens, daily stand-ups, burndown charts, sprints and scrummasters for a second. In my experience all of that weird jargon is really hurting Agile’s adoption among the no-nonsense type of developer that doesn’t have time for fluff and wants to know the real benefits of something before investing time in learning about it (instead of a million other things they could be learning).

I’ve also seen that when anyone happens to cite a failed experience with Agile, the explanation from Agilists is invariably the cryptic “you weren’t doing it right”. This is a bit of an irritating response to anyone that’s spent a significant amount of time following the ceremonies and speaking the lingo though. Instead I’d like to submit that you have a much better chance of succeeding with Agile if you know the real reasons behind the madness.

There are a number of Agile values I’ve noticed in my years of practicing scrum and XP that I think stand on their own even if you’re not doing Agile and just want to improve your development process in some small way. I’m going to list them (and try to stay jargon-free) because I imagine that they would be useful individually, but they do have a sort of self-reinforcing quality to them where their effects will somehow be amplified if you use them together. You should also feel free to break the rules where you feel they need to be broken, but I’ve very rarely found that to be necessary. Usually when I’ve broken these rules, I’ve regretted it later, because what I thought was an exception didn’t really end up being an exception.

Create a team, not just a group of individuals.

A project will move faster when it is being developed by a team that self-organizes around its goals rather than a group of individuals working separately. Here are few rules of thumb to creating a team instead of just having a group of individuals:

Don’t assign tasks to individuals. Let the team figure out who works on what, and let the team take responsibility for all tasks rather than individuals. Everyone on the team needs to agree on what it means to be done and how, and everyone on the team needs to take responsibility for it.
Strive for cross-functional skillsets. Let the team members teach each other so they can better help each other in the future rather than allowing one person to become the bottleneck for any particular kind of task.
Try not to change team size or membership very much. A good team can be a fragile thing.
Throw out the notion of individual code ownership. The entire team owns the code.
Create more of a bullpen environment where the team members are all basically in the line of sight of one another and can just talk to one another about what they’re working on. This was tough, because it made us give up our individual offices, telecommuting, and deciding our own hours, but it was well worth it for the improvement to the quality of our work day. Almost anytime a developer has to use some means of communication other than his voice to talk with a teammate, you have unnecessarily wasted time.
Put QA personnel on the team so that testing can be can be on-going and so the team can more easily take on QA and testing as a team-wide responsibility.
The whole team needs to agree on what kind of work is necessary for each task before it is to be considered “done”. (coding standards, documentation standards, how rigorously they test, etc, etc)
The result of this was a highly capable team that could “swarm” to complete goals reliably and quickly. We actually developed a sort of sports team mentality that made this way of working genuinely fun, and helped us push ourselves to be better all the time. There are a lot of challenges to creating a great team that can inter-communicate openly and honestly, but it’s an amazing thing to behold when it happens.

Estimates need to come from developers.

This is can be very hard for some organizations to swallow, but developers are the ones that are most qualified to do estimations, and as professionals, they should be working diligently to improve their estimation capability. When a developer is responsible for estimation, that developer feels a sense of ownership about his work and will feel a vastly greater sense of responsibility to meet the estimate. Developers that have never learned how to estimate will not be good at it at first, but there’s no other way for them to get good at it than to have them do it. There are a few major things that can vastly help estimation too:

Always break work down into tiny chunks. We generally broke work up into chunks that were less than 3 days of effort. I’ve never seen a task that couldn’t be broken down into subtasks that are this small or smaller, if the developers are so inclined.
The team should take a look at bad estimates in retrospect and figure out where they went wrong. Recognize work that will be impossible to do due to lack of clarity from the business, or unmet dependencies and push to have those requirements met.
For highly speculative work, just commit to doing a proof-of-concept, or time-boxed research.
The more developers that are involved in the estimation process, the better. You’ll probably want to include QA/testers as well, because good ones have unique insights.

Context-switching should be avoided mercilessly.

This means that each developer needs the discipline to make sure he only works on one task at a time, that the team works on as few tasks at one time, and that developers need sustained uninterruptable timeboxes where the priority of work is not allowed to change.

Most people (especially non-developers) have no idea how much of a huge time-suck that context switching is, and few realize how seldom it’s actually worthwhile. Also if the team is tackling as few tasks as possible at once, if any task can be finished by having more developers on it at once, then that’s what the team should do rather than starting as many tasks as possible and working on them in parallel. This might sound counter intuitive, but with the team ultra-focussed on as few tasks as possible at once, they can get the totality completed faster.

Try to always improve.

The team must commit to continually improving. At regular intervals, the team should look at how things are going and pick a few actionable things to make their work better/faster/easier. You’d be surprised at the huge speed improvements that can come from this practice over time. If the team has a shared calendar, set up a reoccurring meeting at a regular interval to have this meeting, and be laser-focussed on deciding what improvement you want to achieve and how exactly you’ll try to achieve that. We’ve always set aside 1 hour per week, but your mileage may vary. Start every meeting by evaluating how successful the team was with the previous meeting’s goals.

If something is difficult or time-consuming, do it more often.

This sounds crazy to the uninitiated but there are two reasons for this:

Doing difficult things more often will drive out new creativity in making them less difficult. Pro-tip: Automation is almost always the answer when the problem is testing (See TDD, BDD, ATDD) or deployment. (See Continuous Delivery)
Many types of tasks get more difficult the longer they are postponed and so are always easier if you do them more often instead of letting the risk of complexity pile up. Integrating code into the main branch is one example (See Continuous Integration). Deploying is another (See Continuous Delivery).

Be merciless about achieving high quality

Over the long haul, speed comes from quality, so everyone on the team needs to make quality their job. Everyone needs to test, and everyone needs to think about how to continually make testing easier without cutting corners.

Have QA personnel on every team, and have them testing as the team works.
The whole team needs to agree on what level of quality is needed, and what quality metrics will be used to judge every check-in.
Perform a root-cause-analysis on each defect. Try to figure out how to avoid that class of defect in the future.

Do everything in tiny increments.

Tiny increments reduce the risk of compounding problems, and allow you to get feedback and correct your course more often.

break-down your work into tiny tasks
plan just-in-time and just enough to do work.
test as you go, instead of in huge batches
merge code to the mainline early and often.
deploy early and often.
don’t add anything to the system that you don’t need until you need it (YAGNI)

Get customer feedback early and often

Get customer feedback early and often to converge on the best possible solution as fast as possible by constantly correcting course instead of blindly following some long-running plan or doing unnecessary work. The customer’s priorities and vision of the product will change as he sees results, so it’s better to not find out about these changes after a lot of unnecessary work has been done.

Do work in priority order as dictated by the customer. This will allow them to see the aspects that they care about most as early as possible, and therefore help you converge on a unified vision of the product faster.
Work break-down is better done in vertical slices where the customer can get some amount of real value from each piece (or at least see some evidence of work) rather than having many non-user-facing tasks, where possible. This allows for more potential for feedback.
The work shouldn’t be considered done by anyone on the team until the customer, or a representative of the customer agrees that it’s done.
Constantly improve estimation skills so that they are generally more reliable so the customer knows the cost of the work he has requested and can make realistic decisions based on that.

Anyway, feel free to try what you want and throw out the rest!

Quality is the Constraint

2010-12-05T22:33:50Z

One fine day in the trenches, a junior engineer was tasked with optimizing a particular web service call that was way too slow (sometimes slower than 20 seconds!). I had just figured out how to use a performance profiler for that particular platform, so I thought I’d offer to show him how to use it. His response was that he was on a tight deadline and didn’t have time to learn to use the profiler, and that he was just going to tune the database calls to make them perform better.

Later in the year when the problem still wasn’t solved, I had a chance to take it on myself. The profiler showed me that this service call was itself making around 20 web service calls to a second system that could obviously add up to 20 seconds, if that other web service took 1 second for each of its responses (and the logs showed that sometimes it did). It turned out that almost all of those web service calls were redundant and completely unnecessary. Simply removing them (which took about 1 day of effort) shaved 19 seconds off this web service call’s duration in production in some cases.

The total execution time spent on database calls was 5 milliseconds. A 50% improvement there would probably take a week’s worth of effort, and really only get you 2 or 3 milliseconds of performance improvement.

The moral of the story is that there’s a pretty simple process for optimizing a system:

Identify and isolate the component that is tying up the most time in the system (let’s call it the bottleneck)
Optimize that component to make it less wasteful
Repeat

Skipping step 1 isn’t a shortcut. It’s absolutely critical to know that you’re optimizing the bottleneck, and not some other random component.

This is pretty obvious stuff for most experienced developers of course. And while they seem to know to apply this to the systems that they work on, it’s more rare that they notice it outside of the code and see it in the process of software engineering in general.

The software development process is a system as well, made up of components. Requirements come in at one end, and deliverables go out the other end. As people who are involved in that process, we naturally want it to be more and more efficient. We are constantly trying to figure out how to optimize that system so we can deliver better software in less time.

So step 1: Identify and isolate the component that is tying up the most time in the system.

Not every team develops software the same way, so this component of the process could be different team by team. In practice though, I’ve always seen teams spending the most time doing these sorts of things:

Refactoring poorly written code so that features can be added
Learning/relearning existing code that has long since swapped out of all developers’ “mind pages” (for extension or bug-squashing).
Debugging
Performing Manual User Acceptance Testing (by both developers and testers)
issue tracking (administration of that tool and issue in it)
discovery and documentation of steps to reproduce
triage (what bugs to squash first?)
communication between customers, support, QA and developers
customer support
customer dissatisfaction leading to churn and reputation problems (affecting the entire organization)

All of these time consumers are affected directly or indirectly by our ability to control quality. In my experience, the majority of the software development process isn’t hung up in requirements analyis, design, or implementation. It’s hung up in all these Quality Control related issues. Quality Control is the constraint that keeps us from creating the right software faster. Steve McConnell confirms it in a more scientific way than I ever could:

The rest of us would do well to learn from a discovery made by IBM in the 1970s: Products with the lowest defect counts also have the shortest schedules (Jones 1991). Many organizations currently develop software with defect levels that give them longer schedules than necessary. After surveying about 4000 software projects, Capers Jones reported that poor quality was one of the most common reasons for schedule overruns (1994). He also reported that poor quality is implicated in close to half of all canceled projects. A Software Engineering Institute survey found that more than 60 percent of organizations assessed suffered from inadequate quality assurance (Kitson and Masters 1993). On the curve in Figure 1, those organizations are to the left of the 95-percent-removal line.

(http://www.stevemcconnell.com/articles/art04.htm — The whole article is extremely illuminating)

What this means is that if we really want to move quickly, we need to take a good hard look at how we manage quality. Of course this is counter-intuitive, and its actually hard to swallow, because ensuring quality can be mundane and tedious if you’re not doing it right. But as developers we’d be silly to ignore the real bottleneck in the process, and optimizing for that. Micro-optimizations elsewhere in the process will yield very little benefit compared to improving measures that ensure quality.

As developers, we love to argue about and discuss these micro-optimizations all day long:

It sure seems silly though to be pursuing micro-optimizations, at the expense of tackling the real #1 time-suck: Quality.

The Lost Art of Prototyping

2009-12-27T00:19:01Z

Prototyping is a design methodology that basically involves “trying stuff out and seeing what works”. It’s meant to be iterative in that you continue to create prototypes (possibly by alterring previous prototypes) until you’ve got all the information you need to say your design is complete.

If we were building cars, we would probably first prototype a body design with CGI to be able to collect feedback before building the car itself. It’s much cheaper (effort and materials wise) to iteratively develop the body with a 3d model than with the actual materials that would be used.

If we were designing websites, we might prototype the design in photoshop for a bunch of iterations until the client is happy. The great thing about that is that the web designer can actually chop up that protoype in photoshop, and use it for images as part of the work after the client has approved the design stage. In this case, the prototype isn’t a throw-away prototype — it’s actually useful to bring into the implementation stage.

In software development we can (and usually do) prototype similarly. There’s usually a point in the design process where a developer needs to “try some stuff out and see what works”. Once the developer has proven a methodology with code, the prototyping phase can mostly be called complete. Throwing away that code would widely be considered a huge waste though, so it usually ends up being used in the final implementation as well.

Prototyping VS. Hacking

It’s probably safe to say that many developers do prototyping (a design methodology) and implementation at the same time bouncing back and forth between the two as necessary, blurring that line between design and implementation.

This way of developing is seductive, because you get straight to banging on the keyboard, which feels like you’re making progress, but it’s really upstream in the design stage where you get the opportunity to save the most amount of time. If you don’t distinguish between design and implementation, it’s tough to tell which stage you’re in at any given time, and so it’s tough to determine if you’re in the right stage.

You can imagine how wasteful this back-and-forth would be if you were an automative engineer instead, and you swapped back and forth between prototyping and working on the implementation details. As you iterate on the prototype, those details you worked on in the last iteration could be completely removed — that’s a big waste of time, and it’s often emotionally difficult to let go of design decisions that have undergone a lot of implementation work. So while realistically design never really stops, you do want to do as much up front as possible.

So there are definitely right and wrong ways to prototype, but it’s extremely valuable and worthwhile if done correctly. I doubt many developers would argue that sometimes you just need to try stuff out and see what works. Very few developers are interested in development methodologies that omit the ability to prototype.

So What about TDD?

This seems to me to be the most often used argument against Test Driven Development (TDD). I agree that TDD doesn’t work well for tasks that don’t have clear-cut solutions, but TDD is still always applicable downstream after the solution has become more clear. The trick is to reach a certain level of solidity with your solution’s plan before you get into TDD.

Here’s a case in point as Ron Jeffries attempts to “solve sudoku” with TDD without even knowing the rules of the game. If you follow his method of solving the problem and compare it against Peter Norvig’s design-first approach, TDD starts to look like a miserable failure. Indeed, some have dragged this out as a case against the utility of TDD “in the real world”. Ron Jeffries’ example doesn’t really make the case against TDD at all though; It simply shows that you have no business being in the TDD stage before you fully understand the problem and you’ve done adequate design. TDD doesn’t preclude other design methods (like prototyping) at all.

And in cases of large architectural issues, prototyping is likely not the first tool that you should turn to — it’s too detail oriented, and you can iterate much faster on designs expressed via whiteboard doodling, flowcharts, UML, etc. You might move to prototyping at some stage later though before you completely decide on some particular design. The important part to keep in mind though is that you’re not in the implementation stage — you’re planning. Changes are cheap and easy at this stage, and it saves you so much more time than if you just jumped in and started developing.

The software that comes out of the prototyping process is a nice side effect, but it’s almost never ready for production. This is where TDD comes in. If you have a clear understanding of the solution you want to implement (whether you used prototyping or not) then your design is sufficiently complete to start TDD. In this way TDD and prototyping are not opposed at all. Prototyping is just one possible methodology in the design stage that should precede implementation. TDD just says that tests should be written before implementation. It doesn’t say anything about what else you do prior to implementation.

So, there’s nothing un-TDD about prototyping a solution, setting it aside, writing your tests, then writing an implementation that passes those tests (based on the knowledge you got from your prototype). Ideally you’re working with units that are single-minded and small so copying/pasting from the prototype isn’t even necessary, but to be honest, I think some copy/paste from the prototype is fine. If it’s done in a careful way where you’re not hurting the quality of the final code (because prototypes generally don’t care all that much about code quality), you might as well make use of your previous work, if you can. You just have to be careful that it’s the quality you want, but then again TDD will force a lot of the quality.

The Steel Thread

The concept of the steel thread is one that I use quite often on the team that I work in when we’re working on complex tasks. It’s definitely a loose sort of prototype, and it’s meant to prove the basics of our core concepts. Basically, we concentrate on the core functionality of the complex task, putting priority on the completion of the “happy path“, the path of execution without error scenarios, special cases, etc. Once that core functionality is in place, the general design can be considered verified, the necessary components can be easily identified, and the existing work can be evolved to something high-quality (possibly through a fresh start with TDD). Work is easily divided amoung teammates after this steal thread has been acheived.

Spike Solutions

Spike Solutions are like the opposite of the steel thread in many ways. Where the steal thread tries to prove the viability of the entire solution (in a breadth-first sort-of-way), a spike solution will try to prove the viability of a part of the solution that isn’t very well known, in a depth first sort-of-way. It’s useful when there’s a particular piece of the solution that we’re not sure will work, so we prototype that part as quickly as possible to make sure our overall solution won’t fail because of that single part.

The point of the concepts of steel threads and spikes is to tackle the unknowns early on and prove the viability of the entire solution as early as possible. Once everyone is relatively sure that the team is on the right path and the design is firmed up, you’re ready to get into the more detail-oriented aspects of development. It’s not until this point that I’d switch to TDD, and start writing my tests for the implementation stage of the work.

Is this just BDUF?

I don’t think so. I’m not proposing a huge documentation process that specifies every aspect of the final product. I’m simply talking about spending some time to loosely figure out what we’re going to do before we do it. It’s time well spent, because it’s just so much cheaper to make major changes at this earlier stage in the process. We’ll still figure out the minor details as we go.

The Future-Proofing Trap

2009-12-21T00:27:28Z

“We will encourage you to develop the three great virtues of a programmer: laziness, impatience, and hubris.” — Larry Wall

In this quote Larry Wall makes some great points about what it means to be a great programmer in a zen-like, toungue-in-cheek sort-of-way. Let’s look at one aspect of the Virtue of Laziness.

Every reasonable developer knows that he should write his code to be easily maintained. As part of that work, it’s natural for that developer to want to try to make his code as ready for future change as possible. Taking that desire a few steps further, he might want to write code that solves future needs in addition to the current requirements, so that that code doesn’t even need to be revisited. Unfortunately, that’s a common mistake that leads to a lot of wasted time and less maintainable code.

I’d like to propose a principle of software development that I’ve come to believe:

The most maintainable code is the simplest possible code that meets all the requirements.

This is almost a tautology because simple code is (by definition) easier to understand, so it’s easier to change/enhance/extend. Code that does more than what’s currently necessary is not as simple as it could be though, so it’s not as maintainable given the current requirements.

One might argue that code that does more than what’s necessary can meet future requirements without change, thereby proving itself to be more easily maintained (no maintenance required!), but all of that hinges upon the developer’s ability to consistently and accurately foresee future requirements. Otherwise that developer has:

wasted time developing unnecessary functionality
decreased the maintainability of the codebase

Even when everybody is pretty certain of the future requirements…

Now let’s consider the case where the developer is almost absolutely sure that the requirements will change a certain way, and so the code should be written to handle that scenario as well. Wouldn’t it be better to do that work now? No. It still takes more time and effort (at a time when we know its not necessary), and it’s still less maintainable code until those requirements change (in case other unrelated changes are required). On top of all that, there’s still risk that the requirements won’t change in the anticipated way!

Even when everyone is absolutely 100% certain of the future requirements…

Now let’s consider the case where everyone is 100% certain that the requirements will change in a specific way, and you can write your code to handle that way as well as the current way. In that infrequent scenario, you don’t face a bunch of losing propositions, so it does make a bit more sense. However, I’d still like to pose a question: how much more work is it to do the additional work later instead of now? Truthfully, the amount of effort is virtually identical regardless of whether you do it now or later. That means you can enjoy maintaining the simpler version until the time comes that it needs to be changed to meet new requirements.

This isn’t a new idea I’ve dreamt up: it’s the YAGNI principle, and it’s an essential part of Uncle Bob’s 3 Rules of TDD. It’s also a great way to avoid unnecessarily complex code.

(It’s important to note too that YAGNI doesn’t say to avoid making your work as futureproof as possble. YAGNI just says not to put extra code into making that possible. For example, if you create a class for a UI button that will be used to save some user data in a form, don’t call it “UserDataSaveButton” — just call it “Button”. It’s no extra work and the re-usability possibilities have increased dramatically.)

We’re not engineers building bridges; we don’t work with unchangeable steel and concrete, and we can accomodate changing requirements easily. It’s unrealistic and wasteful to guess that we know what future requirements will come, especially when the potential work saved in the long run is neglible.

Code Slower

2009-12-14T00:32:30Z

Here’s a controversial opinion, but one that I really believe to be true: most of the things we do to produce code faster are actually counter-productive. It sounds crazy, and I would naturally oppose it except that every time I ignore this lesson, experience has a way of reteaching me. This is as much a message to myself as it is to anyone else that might happen upon it (I’m talking to you, viagra comment spammers).

Development Time VS. Maintenance Time

There’s no doubt that the original development stage of a piece of software can be time-consuming. It’s interesting though, when you consider the difference in time spent in development vs. time spent in maintenance. At what point do you cross into maintenance mode?

Let’s face it: you’re actually also in maintenance mode pretty much right away, unless you’re some superhuman that conceives of the whole system at once and doesn’t make any design tweaks (or even typos) while coding. But you’re not. So the development-time VS. maintenance-time dichotomy is a false one. You’re always in maintenance mode.

Invest Early, Invest Often

So if you spend so much time (all your time) in maintenance mode, then it makes sense to do every little change and tweak in such a way that your subsequent tweaks and changes are as easy as possible. We should be spending the majority of our energy optimizing our time spent on maintenance and not our time spent on development.

Now it’s natural that one might say “well why not optimize both?”, and that’s a valid question. There’s no good reason not to try to optimize both, but we have to remember to never ever ever optimize the development process at the expense of the maintenance process, simply because we just spend so much time in maintenance mode.

Some examples

Skipping the planning stage.

This is generally caused by the “Weeks of coding can save days of planning” fallacy. Uncover all the main parts of your design before you code yourself into a corner that requires a near-rewrite to escape.

Writing untestable code or skipping unit-testing

Unit-tests are your insurance that future changes don’t cause non-obvious regressions. They also have a magical way of creating better factored, higher-quality code. If you skip this stage, your code will not be factored as well as it could be, and come maintenance time, you’ll be scared to make changes for fear of breaking something somewhere else.

Writing classes and functions that do a lot

This one is a really seductive one. We want to be consumers of classes and methods that do a lot for us, because higher levels of abstraction are how we get more done in less time. From the consumer stand-point, this makes lots of sense, and it’s actually the best way to improve development speed and maintenance speed at the same time. If you’re also the person writing the class or function that does more though, you have to be very careful to use the highest abstractions possible for that as well, even if you’re writing its inner abstractions as well. What this means is that every level of abstraction does as little as possible, and you build up to the level of abstraction that you want with those thin, thin layers. Classes and functions that try to do a lot instead of relying on other classes and functions are very hard to maintain, simply because there’s so much to do read and understand, and there’s no easy way for the maintainer to skim and decide what piece is relevant, because the pieces are doing multiple things.

Leaving code in a low-quality state

I’m talking about everything involving non-readable and poorly factored code, including bad names, foggy separation of concerns, etc. Code should not be written just for the machine. It needs to be written for the next guy that comes along as well. That next guy could be you. Or a violent psychopath

Using code generation

Code generation is just pure evil, if there’s ever a remote chance a human will ever have to read it or change it. A computer program doesn’t know how to write code that a human will understand, and human understanding is exactly what we need to strive for if we care about making maintainable code.

Understatement of the day: You’re more than just a typist

It’s largely irrelevant if you can touch type at 90 wpm. It’s largely irrelevant if your IDE can do automated refactors. It’s largely irrelevant if your efficiency with Emacs or vim makes you look like Neo in The Matrix. These things are great bonuses, but won’t make much difference to your proficiency (and efficiency) as a programmer if you neglect to first consider the maintainability of your codebase. (Corollary: In the name of all that is good in this world, please don’t name your variables p1, atb, etc, or your functions like atoi() just for the sake of saving keystrokes!)

Time and time again my mistakes remind me: there’s only one way to be more productive over the long haul: work with better, higher abstractions.