Incrementalism

[   tech   ]            

First: a story. Alice and Bob are sent to an earth-like planet and given the task of finding its highest point. Unfortunately, they are initially given only stone-age era technology to work with. The planet is foggy and visibility is only 20 feet or so. Alice and Bob adopt different approaches:

• Bob just follows the local slope he can see with his eyes until reaching the top of the nearest hill. Maybe he wanders randomly a little ways away… but not too far, because what if he couldn’t find his way back to the locally-highest hill he’d just discovered?
• Alice spends a long time reinventing radar technology, then builds a rocket to launch several satellites to perform a mapping of the planet and provide GPS guidance. She also builds an airplane, because it turns out the landing site is an island, and the tallest point is on the other side of a huge ocean. With all this technology in hand, she then flies directly to the highest mountain.

To an outside observer who didn’t know that rockets, radar, GPS, or airplanes were possible, it would look a lot like Alice was just screwing around, not accomplishing much of anything. “Alice, what the hell are you even doing?” Bob would say. “You’ve just got a pile of metal parts… but I got to the top of a very tall hill over there… in fact I’ve made 200 upwards steps in the past month.”

But then, quite rapidly, Alice’s progress would jump well past Bob. She’d hop on a plane and fly across the ocean directly to the foot of the tallest mountain, then keep hiking up and up and up…

Over long enough time scales, the “shortest path” to any goal usually involves doing things that have no obvious connection to that goal. It involves doing things whose progress cannot be measured in any simple way that would be obvious to someone without expertise. And over these longer timespans, the incrementalist approach of making local improvements according to obvious metrics is a colossal failure. (“You can’t get to the moon by piling up chairs”)

Anyway, none of this is too surprising, but now let’s complicate things a little. Suppose Alice and Bob are put in the same situation and given the same goal, but are told that “whoever gets to the highest point in the next month will be rewarded with additional people, supplies, etc”. And if we imagine iterating this experiment, always rewarding the person who makes the most measurable progress, Bob might amass an army of thousands of people, all wandering around climbing little nearby hills! Meanwhile, Alice gets almost no resources even though she could have have made exponential progress and surged past Bob 10000x. Perhaps Alice is even tempted to abandon her ideas about how things can be made much better long term.

The focus on incremental, measurable progress is often dangerous to innovation and bigger progress.

The software industry

The software industry is swimming in incrementalism. Everywhere you look in software, there are things that can be incrementally improved, and people getting paid to make these improvements. That one library you use, it’s missing a feature. That bit of code, it could be rewritten to be faster. That app, it needs an extra widget or feature. That web standard could use a few additional extensions to it. Etc, etc, etc. There are people working on all these things, because obvious progress is rewarded. Billions of dollars are poured into the industry to capitalize on whatever little incremental improvements can capture some temporary market share. And this process plays out over and over again.

In academia, there’s a different sort of incrementalism. The pressure to produce “publishable” results in the short term (due to the tenure system and other factors) leads researchers to be quite conservative in various ways. They don’t work on anything too crazy that has a high chance of failure. They stay largely within the extremely narrow confines of what they already know about. They do modest extensions to existing research.

Real progress doesn’t look like progress at first. It looks like people screwing around. Playing. Trying different things. Building a rocket and a radar system instead of just walking up the nearest hill. Sometimes the best thing to do is to give smart people a very very long leash and plenty of resources, to be patient, and not worry about progress.