Where 99% of My Dataset Went

I set out to build a dataset of a few million training examples. When the pipeline finished, it handed me a couple of percent of that — a fraction so small it looked like a typo. Nothing had crashed. No error had fired. The data had not failed to generate; it had quietly thrown almost all of itself away, and then reported success.

The number that was too small to be right

The strange thing about this kind of bug is that it does not announce itself. A pipeline that crashes is loud. A pipeline that silently produces far too little just finishes, hands you a file, and waits for you to notice. If I had not happened to remember roughly how many examples I expected, I might have trained on the survivors and never known the rest had gone missing.

The only symptom was a count that was off by nearly two orders of magnitude. And a count being wrong is one of the easiest symptoms in the world to skip past, because a smaller-than-expected number still looks like a real number. It does not feel like an error. It feels like a result.

”Why is there so little?” is a real class of bug

We are trained to debug things that break. We are much worse at debugging things that are simply absent — data that should exist and doesn’t, examples that should have been created and never were. There is no stack trace for a road not taken. The pipeline did exactly what I told it to; it just turned out that what I had told it to do discarded most of the input on the way through.

Where the data went

The cause was almost embarrassingly mundane. Somewhere in the process, I was stepping through the timeline in large strides — looking at one moment, then skipping a long gap to the next, then skipping again. Every position I stepped over was a potential example that was never examined and never created. I had set that stride early, for a reason I no longer remember, and then forgotten it was there.

So the dataset was not corrupted, or filtered, or cleaned down. It had simply never been fully looked at. The overwhelming majority of the available data sat one step outside the loop’s stride, untouched, while the pipeline marched past it and reported that it was done.

Silent loss is the dangerous kind

This is the part I keep coming back to. A loud failure is a gift: it stops you, points at itself, and demands attention. A silent loss gives you none of that. It produces output that looks plausible, runs without complaint, and quietly changes the foundation everything downstream is built on. You can train a model, evaluate it, and draw conclusions from it, all on top of a dataset that is secretly a tiny, unrepresentative sliver of what you thought you had.

The danger is not just that the number is wrong. It is that the number is wrong and the system is happy about it — everything green, everything running, everything built on a base that betrayed you at the very first step.

The fix: examine everything, then choose what to keep

The repair was to stop skipping. Instead of striding across the timeline and only inspecting the points I happened to land on, I had the process consider each position in turn — and then made the decision about what to keep an explicit, deliberate filter rather than an accidental byproduct of a sampling step. The data I dropped, I now dropped on purpose, for a reason I could state out loud. The data I kept, I kept because I had actually looked at it.

The dataset came back close to the size it should have been. Nothing clever made that happen. I had simply stopped throwing data away without meaning to.

The cheapest check in data work

The lesson is almost too simple to feel like one: compare the size you expected to the size you got. It is a cheap sanity check that I had badly underused, and it would have caught this in seconds. A large gap between how much data I thought I had and how much data actually exists deserves to be treated as a bug — even, especially, when nothing crashed to tell you so.

Now, the first thing I do with any dataset is not look at its contents. It is look at its count, and ask whether that number is even plausible. Many of the worst data problems I have had did not corrupt the data. They just silently produced less of it than I believed, and let me build on the difference.

A pipeline that loses most of your data without raising its voice is not crashing — it is running exactly as written, and still doing the wrong thing. The bug was never a fault in the machine. It was in the gap between what I assumed and what I bothered to check.

— No signals, no returns, not investment advice.