I Only Studied the Survivors

For a long time I kept drawing confident conclusions from the data in front of me, and for just as long I failed to notice the most important thing about that data: it had already been filtered. Filtered by survival. The cases that mattered most had been quietly removed from the picture before I ever showed up to look at it, and because they were gone, I never thought to account for them.

This is one of those biases that is famous, that everyone nods along to when they hear it, and that almost everyone — including me — then proceeds to fall straight into anyway. Knowing about it and being protected from it turn out to be very different things.

The classic shape of the error

The most well-known illustration of this comes from wartime, and it is worth recalling in the abstract because the shape is so clean. Imagine studying the aircraft that return from missions to decide where to add armor. You look at where the returning planes are riddled with damage, and the obvious move is to reinforce exactly those spots.

It is exactly wrong. The planes you are studying are, by definition, the ones that made it back. The damage you see on them is damage a plane can take and still survive. The planes hit in the truly fatal places are not in front of you to be examined, because they did not return at all. The damage you most need to protect against is precisely the damage you never see — because the evidence of it removed itself from your sample. The survivors point you in exactly the wrong direction.

My version of the same mistake

In this domain, the trap wears different clothes but has the identical skeleton. The things still around to be studied today are, necessarily, the ones that survived. The assets that exist now are the ones that did not go to zero, did not vanish, did not quietly disappear somewhere back in history. If I study only what is currently in front of me, I am studying a population that has been pre-selected for the single trait of not having died.

And then, on the basis of that selected sample, I would draw conclusions about a world that very much includes dying. I was effectively learning the characteristics of winners and treating them as the characteristics of participants, as though the two were the same group. They are not. The difference between them is a graveyard I was not looking at.

The dead do not show up in the data

The genuinely insidious part is the mechanism. The failures remove themselves from your dataset. Things that blew up, that got delisted, that went to nothing and were forgotten — they are simply not in front of you anymore. And because they are not in front of you, they exert no weight at all on your conclusions, despite being the single most important body of evidence about how things actually go wrong.

So your data has been quietly curated, and curated by exactly the outcomes you most need to understand. The disasters edit themselves out. What remains is a tidy collection of things that, one way or another, made it — and a tidy collection of survivors is a deeply misleading thing to study if your real question is about risk, because risk is precisely the story of the ones that did not make it.

Optimism is the default output

What makes this especially dangerous is that the bias has a consistent direction. It does not scatter your conclusions randomly; it pushes them, every single time, the same way. Survivorship bias always makes things look safer, more profitable, and more reliable than they truly are, because the catastrophes have been removed from view before you began.

This means any analysis built on survivors is systematically rose-tinted, and the tint is nearly impossible to detect from the inside, because you cannot see what is missing. Absence does not announce itself. An empty space where a hundred failures should be looks exactly like an empty space. Nothing in the data you have will ever spontaneously tell you about the data you do not have. You have to know to ask.

You cannot analyze what is not there

This is why the fix is so counterintuitive, and why being smart about your available data does not save you. The problem is not in the data you have; it is in the data you do not have. No amount of rigor applied to a biased sample removes the bias, because the bias does not live in the sample — it lives in the gap between the sample and reality. You can analyze survivors flawlessly and still be completely wrong about the world, because the survivors were never the whole world.

The only real remedy is to actively go and find the dead. To deliberately seek out the failures, the things that no longer exist, the cases that removed themselves, and to put them back into the picture by hand. This is genuinely harder, because they hide — that is the whole nature of the problem. They do not come to you. You have to go looking for absence, which is an unnatural thing to do.

Counting the silence

The practical discipline that came out of this is a question I now try to ask constantly: what is missing from this picture, and why is it missing? Whenever a dataset looks clean and encouraging, the right reflex is not to feel encouraged. It is to ask whether it looks that way because reality is genuinely encouraging, or because the discouraging cases were quietly removed before I arrived to look.

Learning to notice and count the silence — to give weight to the things that are conspicuously not there — turned out to be one of the more important habits I built. It is a strange skill, paying attention to absences, because absences are by their nature easy to overlook. But in a world that edits out its own failures, the ability to feel the shape of what has been removed is most of the battle.

The deeper lesson: the absent evidence is the most important

What I ultimately took from all of this is that the most valuable evidence is very often the evidence that has been removed, specifically because it represents failure. The graveyard contains far more information about risk than the winners’ circle ever will — but the graveyard is silent, and you have to choose to walk into it, because it will never call out to you.

So now, when I see an encouraging pattern, my first question is not “how encouraging is it?” It is “who and what had to disappear for this to look as good as it does?” The picture that survives to reach you has, almost always, been edited by the very outcomes you most need to study. The hardest and most important work is reconstructing what was cut.

— No signals, no returns, not investment advice.