Where should you armor the bomber — where the bullet holes are, or where they aren't?

During World War II, the damage on bombers that made it home looked like a guide to where armor was needed. A statistician named Abraham Wald saw what that data was missing: the planes that never came back.

What came home: damage on returning planes

A four-engine bomber with bullet holes on the wings, fuselage, and tail, and none on the engines or cockpit
Bullet holes found on returning planes

What Wald found: the areas without holes

The same bomber with the engines and cockpit highlighted, the areas with no holes on returning planes
Most vulnerable: engines and cockpit

The bomber and bullet holes are illustrative, not historical damage counts.

Where the holes were Wings · Fuselage · Tail
VS.
Where the planes were most vulnerable Engines · Cockpit

The safest-looking spots on the plane were actually the most dangerous ones. A dataset built entirely from survivors can't show you what killed everyone else.

The same trap shows up anywhere the failures drop out of the data: interview only employees who stayed, and you miss why people quit; study only companies that succeeded, and you miss the ones that took the same risks and failed.

What had to survive to be counted in the data you’re looking at?

Why does the armor go where the holes aren't?

The rule: the obvious reading is to armor where the holes are, since that's where the damage is concentrated. But every plane engineers could inspect had already survived its damage. A hole in the wing or tail was evidence that a plane could take a hit there and still fly home. The spots with almost no holes, like the engines and the cockpit, weren't lucky. Planes hit there were more likely to go down and never make it back to be counted.

Wald, working with the Statistical Research Group at Columbia University, developed a method for estimating how vulnerable each part of a plane was using only the damage on planes that survived. His analysis found the engine area especially vulnerable: hits there were much less likely to show up among the planes that returned. The popular retelling adds that his advice was adopted and saved many crews, but the historical record of what happened next is thin.

It is the same selection effect, working in the opposite direction, as our Regression to the Mean exhibit: survivorship bias hides the data that never made it back to be counted, while regression to the mean over-weights whichever data point happened to land on the extreme end of ordinary luck.

Related exhibits

Sources: Abraham Wald, A Method of Estimating Plane Vulnerability Based on Damage of Survivors, Statistical Research Group, Columbia University (WWII), reprinted by the Center for Naval Analyses, 1980; "The Legend of Abraham Wald," AMS Feature Column. The bomber drawing and bullet-hole pattern are illustrative, not Wald's data.