The Night My Computer Rebooted Itself

One night, in the middle of a long job, my whole computer turned itself off and came back on. Not the program — the machine. One moment a process was running; the next I was looking at a fresh login screen, as if someone had pulled the power cord and pushed it back in.

I had spent days being careful about the one resource I was certain mattered. The crash came from the one I had never thought to watch.

The crash with no message

The most unsettling part was how little there was to read. No stack trace, no exception, no log line marking the end. Just an abrupt reboot and, buried in the operating system’s own records, the digital equivalent of a shrug: the system restarted without shutting down cleanly.

A normal bug leaves a body. This one left an empty room. And an error with no message is its own kind of clue — it usually means the failure happened somewhere beneath your program, in a layer that does not stop to explain itself before it takes everything down with it.

I had budgeted the wrong memory

I was running a very large batch of work on the GPU, and I had sized that batch with care — against the GPU’s memory. I knew how much VRAM the card had, I knew roughly how much each unit of work would take, and I had done the arithmetic to stay safely under the ceiling. As far as I was concerned, memory was a solved problem.

It was solved for the wrong memory. The card’s VRAM was fine the entire time; if I had been watching its meter at the moment of the crash, it would have looked perfectly calm. The ceiling I actually hit was the host machine’s system RAM — the ordinary memory the rest of the computer runs on — and I had not budgeted that at all, because in my mental model it simply was not the constraint.

Where the host RAM went

Sending a huge batch of work to a GPU is not free on the host side. Before the GPU can run anything, the work has to be prepared, set up, and handed across — and that preparation lives in system RAM. Each piece of that setup was modest on its own. Multiplied across a batch large enough to saturate the GPU, the modest pieces stacked into something enormous.

System RAM filled. The operating system did what it does when memory runs low — it spilled over onto disk, and then ran short of that too. Severe memory pressure does not always announce itself with a catchable exception. It can mean a failed allocation, a killed process, or a whole machine turning unstable. Mine turned unstable hard enough to reboot.

Why I never saw it coming

The frustrating thing in hindsight is that I was monitoring carefully — I was just monitoring the wrong gauge. My eyes were on the GPU, the resource I had optimized and worried over and felt clever about. The whole time, the pressure was building in a place I had no reason to look, because I had already decided it was not where the problem would be.

That is a pattern I have now seen too many times to call it coincidence: you instrument the constraint you are thinking about, and you go blind to the one you are not. The bug does not hide. It sits in a blind spot you built yourself, by being confident about where the limit was.

The fix: expect the wall, and bounce off it

The repair was not a better calculation. It was giving up on the idea that I could calculate the safe size in advance at all. Instead of predicting exactly how large a batch the machine could survive, I let the machine tell me: attempt the batch, and if it runs into a recoverable memory error, treat that as a signal to back off — reduce the workload conservatively and retry, shrinking until it fits.

It is a humble little loop, and it is far more robust than my careful arithmetic ever was, precisely because it does not depend on my having understood every hidden cost in advance. I no longer need to know exactly where the wall is; for recoverable memory errors, the process can feel its way to a workable size on whatever hardware it runs. Host-level exhaustion like that night’s is the harder case — it still needs external limits and monitoring, not just a retry loop.

Resource assumptions are silent until they aren’t

The lesson I kept has little to do with GPUs specifically. It is that resource assumptions are invisible right up until the moment they are catastrophic. You can be exactly right about the limit you are picturing and still be undone by the one you are not — and the failure, when it comes, often arrives with no explanation, because the layer that breaks is below the layer where your error messages live.

So I stopped trying to be precise about resources and started trying to be defensive about them. Precision assumes you know all the costs. Defensiveness assumes you have missed at least one. In a system meant to run for hours, unattended, on memory you do not fully control, the second assumption is the only safe one.

A program that crashes leaves you a message. A machine that reboots leaves you a silence. I have come to treat that silence as one of the more honest things my hardware has ever told me: you were budgeting the wrong thing.

— No signals, no returns, not investment advice.