I Lost a Week to a Crash I Should Have Survived

I once lost the better part of a week’s work to a single crash. Not because the crash itself was some rare catastrophe — it was an ordinary, boring interruption, the sort of thing that happens all the time. I lost the week because I had saved nothing along the way. So an interruption that struck a few minutes from the end threw away days of accumulated progress, and there was no way to get any of it back. It was simply gone, and it had been gone-able the entire time without my noticing.

The painful part, in retrospect, is how completely avoidable it was. The crash did not destroy my work. My failure to make my work survivable did.

The quiet optimism of assuming it will finish

I had built my long-running processes as though they would always run cleanly from start to finish, uninterrupted, every time. No saving of intermediate state. No way to recover partway through. Just kick it off, let it run, and assume it would reach the end. If you had asked me directly whether that assumption was safe, I would have said no. But I had never asked, and so I kept building on it anyway.

This is a pervasive and almost invisible kind of optimism: quietly assuming the happy path is the only path. You design for the world in which everything goes right, because that is the world you are imagining while you build. And then you ship it into the actual world, which has many other paths, most of them less pleasant.

Anything that runs long enough will be interrupted

Here is the truth I had refused to plan around. Any process that runs for long enough will, eventually, be interrupted. A crash. A power blip. A machine that runs out of memory. A mistake on my part. A reboot that could not wait. The specific cause does not matter, and there are more causes than you can enumerate.

Over a short window, interruption is a risk. Over a long enough horizon, it stops being a risk and becomes a certainty. It is not a question of whether a long-running thing gets interrupted, only when and how often. And once you see it that way, designing as though it will not happen is revealed for what it is: planning, deliberately, to be wrong about something guaranteed to occur.

The cost is set by what you saved, not by the crash

This is the insight that reorganized how I thought about all of it. The damage from an interruption is determined almost entirely by how recoverable your work was — not by how severe the interruption was. The crash is rarely the variable that matters. What you did beforehand is.

The exact same crash, striking at the exact same moment, costs you five minutes if you had saved your progress recently, or five days if you had saved nothing at all. The interruption is identical in both cases. The consequence is wildly different, and the difference was determined entirely by a choice you made earlier, long before the crash arrived. The crash does not decide how much you lose. You do, in advance, by deciding how much you were willing to make recoverable.

Checkpointing is arithmetic, not paranoia

Once I framed it that way, the case for saving intermediate state regularly stopped looking like excessive caution and started looking like simple arithmetic. Interruption is certain over time. The cost when it happens is bounded by how long ago you last saved. So saving frequently is just cheap insurance against a guaranteed event, with a premium you can set as low as you like by saving a little more often.

Refusing to do it is not bravery or focus or moving fast. It is just bad math — declining inexpensive protection against something certain to happen, in exchange for a small convenience now. I had been congratulating myself on not wasting time on safety nets, while quietly running an enormous, uninsured risk against an event that was not even in doubt.

Design to resume, not just to restart

There is a level beyond periodic saving, though, and it is the one that mattered most. The deeper fix is to build long-running work so that an interruption resumes from where it stopped, rather than starting over from the beginning.

This distinction is everything when interruptions are frequent. A process that restarts from zero every time it is knocked down can, in a bad enough environment, never finish at all — each interruption erases everything, and if they come often enough relative to the runtime, you make no net progress no matter how hard you work. A process that resumes, by contrast, always moves forward. It can be knocked down again and again and still, eventually, arrive, because each interruption costs only the little bit since the last save. Restarting is fragile in a way that compounds; resuming is robust in a way that compounds the other direction.

The same principle, almost everywhere

This generalizes far beyond any one long-running computation. It applies to nearly any work whose value accumulates over time and is lost if it is not deliberately preserved. The pattern is identical: progress builds up, an interruption arrives, and whatever was not made durable disappears.

So the question I learned to ask, about anything that takes a while and matters, is no longer the comfortable one — “what if this gets interrupted?” — held at arm’s length as a hypothetical. It is the blunt, assume-it-will-happen version: “when this gets interrupted, how much will I lose?” That question, asked early, changes what you build. The first question lets you hope. The second forces you to prepare.

The deeper lesson: the interruption is a scheduled event

What it all comes down to is that I stopped designing for the world where everything goes right, and started designing for the world where things routinely go wrong — because the second one is the world that actually exists, and the first one was never anything but a pleasant assumption I had quietly substituted for reality.

Resilience, I came to understand, is not pessimism. It is accuracy. The interruption is not a remote hypothetical to be optimistically discounted. It is, in effect, a scheduled event — you simply do not get told the date. Building as though it will come is not gloomy or fearful. It is just honest about the only world there is to build in. I lost a week learning that the crash was never the problem. The problem was that I had built something with no way to survive being interrupted, in a world where being interrupted is guaranteed.

— No signals, no returns, not investment advice.