The Bug Was How Hard I Searched

For a while, I measured my own seriousness by the size of my search. I tested hundreds of millions of configurations of a strategy, and then, when that felt insufficient, billions. I was proud of the number. It felt like rigor — like I was leaving no possibility unexamined.

It was closer to the opposite of rigor. The sheer size of the search was not helping me find an edge. It was manufacturing a mirage, and the bigger I made it, the more convincing the mirage became.

The pride that should have been a warning

I used to mention the scale of my search the way you might mention how many hours you worked: as evidence of effort, of thoroughness, of taking the problem seriously. Hundreds of millions of combinations evaluated. Then more. There is a real satisfaction in pointing at a number that large and saying, in effect, look how exhaustively I looked.

The satisfaction was the problem. I had quietly mistaken the size of the search for the quality of the result, and those two things are not merely unrelated — past a point, they pull in opposite directions.

What a giant search actually does

Here is the uncomfortable mechanism. Take any fixed slice of history and try enough different strategies against it, and some of them will look spectacular — not because they are good, but because, out of millions of attempts, a few are bound to line up with the past’s particular accidents by sheer luck. The more configurations you test, the more certain you become of finding one of these flukes, and the more extreme the luckiest fluke will look.

So a giant search does not reliably surface the best real strategy. What it reliably surfaces is the best-fitting coincidence — the configuration most precisely shaped to the random quirks of the exact data you searched over. And it presents that coincidence to you wearing the most beautiful results you have ever seen.

This trap has a name

None of this is new. It is one of the most well-worn traps in all of statistics, with a small museum of names: multiple comparisons, data dredging, p-hacking. The core of it is always the same. Test one hypothesis, and a strong result carries information. Test a billion against the same data, and the probability of finding an apparently exceptional result by chance alone can become very high — whether or not anything real is there. With enough tries, “exceptional by luck alone” stops being a long shot.

I was running a billion-hypothesis experiment and reading the winner as if it were a single, hard-won discovery. It was not. It was the loudest noise in a very large room.

Why it felt like the opposite of overfitting

The cruel part is how virtuous it felt. “Leave no stone unturned” is supposed to be good advice. But leaving no stone unturned against a single fixed dataset is one of the surest ways to guarantee that you will eventually turn over a stone that happens, by chance, to be shaped like treasure. Thoroughness against one sample is not the same as rigor. Often it is just buying yourself more and more chances to be fooled, and then being impressed when one of them pays off.

I thought I was being careful. I was being careful in the one direction that made things worse.

The tell: the winner never survived contact

The signature was usually the same, once I learned to look for it. The configuration that won the enormous search looked incredible on the data it had been selected against, and then fell apart soon after it met data it had not been chosen on. That collapse may not be mere bad luck. It is a common warning sign that the search found a coincidence — strong in-sample, because in-sample is exactly what it was optimized to fit, and weaker out-of-sample, because the coincidence may not reliably repeat. (A real shift in conditions or a plain bug can cause a collapse too; selection on a huge search is just a prime suspect.)

A mitigation: constrain the search, protect the validation data

The repair had two halves. The first was to stop treating the size of the search as a virtue — to run smaller, more deliberate explorations driven by some reason to expect a configuration to work, rather than vast blind sweeps that simply mine the data for whatever fits. The second, and more important, was to insist that any winner show evidence of robustness on data it was never allowed to see while it was being selected — knowing that even a clean holdout, consulted too often, slowly stops being clean.

That second rule is much of the game. The question that matters is not “what is the best-looking thing I can find in the past?” Anyone with enough compute can find a great-looking thing in the past; the past is finite, and a big enough search will eventually fit its accidents. The question that matters is “does the best thing I found survive a stretch of data it could not possibly have been shaped to?” Almost everything my giant searches turned up failed that question, and the failure was among the most useful results they ever produced.

More compute is not more truth

The lesson I keep is a quiet correction to an instinct that feels almost irresistible in this field: that if you are not finding an edge, you simply need to search harder — more configurations, more compute, more brute force. For some problems that instinct is right. For the problem of finding a real, durable edge in noisy data, it is often closer to backwards. More brute force does not necessarily bring you nearer the truth. Sometimes it just gives the noise more entries in the contest — and noise, given enough entries, tends to win something.

One of the most honest things a search ever did for me was come back, after an enormous number of tries, with nothing that survived. It took me an embarrassingly long time to understand that this was not a failure of the search. It was the search, finally, telling me the truth.

— No signals, no returns, not investment advice.