The Evolution of Agile Imposters

Or, Why We Can't Estimate Our Own Software And How That Hurts Us

Put simply, programming is fundamentally misaligned with the way we evolved to estimate effort.

Our brains evolved in situations where the amount of work required to execute on a solution scaled pretty much linearly with the size of that solution. When you're building a mud-brick hut, building twice as big takes almost exactly twice as long. When you're picking berries, picking twice as many berries takes twice as long. When you're out hunting, hunting twice as many deer takes twice a long. When you're farming, planting twice as much wheat takes twice as long.

Software, in contrast, offers solutions whose sizes are almost uncorrelated with the effort that went into them.

On the difficulty side, software is one of the all-time greats. We have mind-bending mathematics, we have the interaction between complex logic and the human condition, we have solutions that must be maintained in a changing world, we have everything that makes math hard and everything that makes creativity hard and everything that makes society hard.

As for size, code is intentionally designed to abstract away all of the decisions that led up to it. Efficiency and portability require that the requirements-gathering and social decision-making and politics and compromises and optimization and tweaking and tuning and thinking and testing are stripped from the finished artifact. Opinions become meetings become designs become source become binaries become behavior. Encapsulation and decoupling are watchwords. The computer doesn't care why you wrote an instruction, only what instruction you wrote. The ideal code is itself and nothing but itself.

Put these extreme pressures together and you get artifacts whose costs are wildly out of proportion to their size. I have seen single flag changes that took literally tens of thousands of person-hours of argument to nail down. I have seen individual numerical constants whose values cost millions of dollars to find and encode optimizations that save tens of millions of dollars annually. status.proto is five lines long, 0x5F3759DF is 32 bits, the constants that control the descent and ascent of the Apollo 11 Lunar Module fit on one screen, the tz database has about six thousand non-comment non-blank lines it, the list goes on and on.

The non-relationship between size and effort means that human brains are simply not equipped to intuitively estimate how much work went into a piece of code. We look at our miserable little pull request and our brain thinks, "Wow, that took me a whole week?" Never mind that nobody else on Earth had the context and experience for it and that we spent the whole week groveling through half-corrupt binary log messages and poking bytes into an MCU's living firmware, our brain looks at the single bit that we found out of place and intuitively estimates the difficulty of flipping it as if it was a basket with a single berry in it. Or worse, given that we can't hold code in our hands or put it in our mouths. And then our brain concludes that we're terrible at programming and that the next bugfix should only take us one day. It is wrong. But it still does it, because it evolved to do it, and we can no more stop it from doing it than we can will ourselves to stop seeing Kanizsa's triangle.

What we can do is understand what our brain does and consciously compensate for its errors. I tell this to my friends when they say things like "Why are people using my library? It doesn't really do anything." People don't use your library because it's a million lines of code or whatever. They don't use it because banging it into the keyboard required seventh-dan martial arts colemak mastery. They use it because those thirty simple lines of code are the distilled wisdom of a unique lifetime of experience and days, weeks, or months of careful consideration and care and understanding. That's what your library means, and it should mean that to you just as much as it does to your users.

If you don't do software, note that this problem isn't unique to programming! Other contenders for the crown of "least correlation between time invested and size of resulting product" include academic research and mental health practice, both of which also have high rates of impostor syndrome. The PhD student's brain says it's "just one equation" and the psychiatrist's brain says "all I do all day is talk to people." Don't let your brain say that kind of thing to you, whatever it is you do.