The Week I Spent Chasing a NaN

One day my training loss stopped being a number. Not high, not unstable — just NaN, the little flag a computer raises when it has been asked to do arithmetic that has no answer. And then it took me the better part of a week to find out why, because the place the symptom appeared was nowhere near the place the problem actually lived.

The most uninformative failure there is

A NaN is, in a way, close to the worst possible error message, because it is barely a message at all. A crash at least points at a line. An exception carries a type and a trace. A NaN just quietly poisons whatever it touches and keeps going: add it to anything and you get NaN, multiply it by anything and you get NaN, and with ordinary floating-point aggregation, average a million perfectly good numbers with one NaN among them and the whole average comes out NaN.

So by the time it surfaced in my loss — loud, final, training-stopping — it told me almost nothing about where the actual bad value had entered. The symptom was about as far as it could be from the cause, with a long chain of arithmetic in between, every link of which faithfully passed the poison along without complaint.

Debugging a NaN is archaeology

In this case, there was no clean way to debug it forwards. The NaN you can see is the end of a story; the bad value that started it happened earlier, somewhere upstream, and has since been laundered through layer after layer of computation. So you work backwards. You add checks at each stage — is the value finite here? here? here? — walking the pipeline in reverse, looking for the first place a real number became an un-real one.

It is slow, unglamorous work, and it feels less like programming than like excavation: digging back through strata of transformations to find the exact layer where things went wrong. A week of my life went into that dig.

The culprit: a transformation applied twice

What I eventually unearthed was almost insultingly small. Somewhere in my preprocessing, a transformation was being applied to a value — and then, because of a refactor I had done weeks earlier, applied to it a second time. Each application was perfectly correct on its own. The first one did exactly what it was supposed to. The second one took the already-transformed value and pushed it somewhere the math could no longer follow, into a region where the operation simply has no defined result. Out came something that was not a number, and off it went, downstream, to surface a week later as a dead loss.

The bug was not a wrong transformation. It was a right transformation that ran one time too many, because two different parts of the code each believed they were responsible for it.

Why preprocessing breeds exactly this bug

This kind of thing is not bad luck; it is what preprocessing pipelines tend to produce when you let them sprawl. Data preparation has a way of getting smeared across a codebase — a little reshaping here, a little rescaling there, applied in different places by different functions written at different times. And the moment the same transformation can be reached by two different paths, the risk grows that some value will travel both of them and get transformed twice.

Nothing about either path is wrong in isolation. That is what makes it so hard to see. The bug does not live in any single piece of code; it lives in the relationship between two pieces, neither of which knows about the other.

The fix: one place, one time

The repair was not a patch to the bad line. It was structural. I pulled the transformations out of the scattered places they had accreted and put them into a single, central stage with one clear rule: each value is transformed here, exactly once, and nowhere else. After that, there was no second path for anything to travel. The specific NaN disappeared, but more importantly, it reduced the number of paths where inconsistent application could happen at all — not merely fixing this bug, but shrinking the space in which its siblings could appear.

That is the kind of fix worth chasing. Patching the one bad value would have made this NaN go away. Centralizing the transformation made many future versions of this NaN much less likely too.

The real lesson: mind the distance between cause and symptom

What made this bug expensive was not its difficulty. It was the distance. The failure showed up in one place, the cause sat in a completely different place, and nothing visibly connected them except a long, silent chain of math that dutifully carried a bad value from one end to the other.

The defenses that actually help against that kind of distance are not cleverness. They are structure and checks: centralize the dangerous operations so there is only one place for them to go wrong, and validate values at the boundaries — check that a number is finite close to the moment it is produced, not a week later when it finally detonates. A bad value caught where it is born can cost you minutes. The same bad value caught where it explodes can cost you a week.

— No signals, no returns, not investment advice.