The bullet holes on the planes that came home

Opening

A bomber comes back from a mission with holes punched through its wings and fuselage.

Another returns with the same pattern.

Soon you have a map of where surviving aircraft tend to get hit.

Where should you add armour?

The famous answer is: not where the holes are.

But the story behind that answer is more interesting than the version that fits on a motivational poster.

The aircraft you can inspect

During the Second World War, the US military drew on the Statistical Research Group at Columbia University to tackle problems with mathematics. One of its members was Abraham Wald, a formidable statistician who had fled Europe.

Aircraft vulnerability presented a nasty data problem.

Engineers could inspect the bombers that returned from combat. They could record where those aircraft had been hit.

What they could not inspect were the aircraft that had been shot down.

That means the sample was selected by survival.

A dense patch of damage on returning aircraft does not automatically identify the place most in need of protection. It may identify a place where an aircraft can take a hit and still make it home.

A part with few holes might be safer from enemy fire.

Or hits there might be the reason those aircraft are missing from the hangar.

The meme and the maths

The modern version often adds a satisfying scene.

Military officers supposedly point to the bullet-riddled wings and tell Wald to armour them. Wald dramatically tells everyone they have it backwards: armour the untouched engines because the planes hit there never return.

It is a cracking story.

The historical record is less theatrical. Wald's surviving memoranda show a serious mathematical effort to estimate aircraft vulnerability from incomplete data. Historians of the episode have noted that the popular confrontation with baffled officials appears to be an embellishment.

That correction actually improves the story.

The problem was not that everyone except Wald was an idiot who had forgotten crashed aircraft existed. The difficult part was working out what could legitimately be inferred when a large and important chunk of the data was permanently unavailable.

Wald developed methods for doing exactly that.

The empty space in the dataset

Survivorship bias now has a life far beyond bombers.

Study successful companies and you can miss the thousands that used similar strategies and failed.

Ask long-lived people what they ate and you are interviewing people who survived long enough to answer.

Look at famous artists who ignored advice and you do not see the equally stubborn artists whose work disappeared.

The returning aircraft make the idea beautifully physical.

There are the holes.

There are the wings.

There is the metal you can touch and count.

And then there is the most important part of the picture: a collection of aircraft that are not there.

Perhaps that is why the simplified Wald story refuses to die.

Even the story about missing evidence has acquired details that are easier to remember than the evidence for them.