My GPU Was Slower Than the CPU It Replaced

I once moved a heavy computation from the CPU to the GPU, expecting it to fly. Instead, it crawled — dramatically slower than the ordinary code it was supposed to replace. I had bought speed and somehow installed a slowdown.

The hardware was not the bottleneck. A modern GPU is one of the fastest things you can put in a computer. The problem was that I had given it work shaped in a way it is almost uniquely bad at, and then been surprised when it could not run.

The upgrade that was a downgrade

The expectation was simple and, I thought, safe: GPUs are fast, my job was slow, so moving the job onto the GPU should make it fast. That is the folk version of how acceleration works, and it is wrong in a specific, instructive way.

When I measured the “accelerated” version, it was not a little slower. It was slower by a large multiple — the kind of result that tells you the problem is not a missing optimization but a fundamental mismatch. Something about the shape of what I was doing was actively fighting the hardware.

What “putting it on the GPU” actually meant

Here is what I had really done. I had an existing loop: take one unit of work, compute it, move to the next, thousands of times over. To “use the GPU,” I had simply made each individual unit run on the GPU — and left the loop exactly where it was.

So thousands of times in a row, my program was turning to the GPU, handing it one small task, waiting for the answer, and turning back for the next. The loop was still in charge. The GPU was just a very fast contractor I was phoning, individually, for every single nail I wanted hammered.

The hidden cost: it costs something just to ask

Every time you hand a piece of work to a GPU, there is a fixed cost to set it up and dispatch it — small on its own, invisible if you pay it once. I was paying it thousands upon thousands of times. When I looked closely, most of the runtime was not computation at all. It was overhead: the GPU sitting idle, waiting to be told what to do, while my program spent its time asking one tiny question after another.

I had built a process that was almost entirely the act of asking, with the actual work as a rounding error. The CPU version had been slow, but at least it stayed busy. The GPU version was slow because it spent most of its life waiting.

A GPU is a bus, not a taxi

The mental model that finally fixed it: a GPU is not a faster way to do one thing. It is a way to do an enormous number of the same thing at once. It is a fifty-seat bus, not a sports car.

Calling it once per item is like running that bus back and forth across town with a single passenger each trip. The bus is genuinely fast. Your throughput is dismal — and it is dismal precisely because you are using a vehicle built for crowds to move people one at a time. The fault is not in the bus. It is in the dispatching.

The fix: stop looping, start batching

The repair was not a faster kernel or a better card. It was restructuring the loop. Instead of “for each item, send it to the GPU,” I reshaped the work so the GPU could take many items at once and process them together — turning thousands of tiny dispatches into a handful of large ones. The overhead I had been paying over and over collapsed into something I paid a few times instead of thousands.

This is one common form of what people mean by “vectorizing” or batching: expressing many similar operations together instead of as a long sequence of small steps. It is often the difference between a GPU that embarrasses your CPU and a GPU that is embarrassed by it, and the gap between the two can be enormous.

Why this is the whole point of the hardware

It took me a while to internalize why the structure mattered so much, rather than treating it as a tuning detail. A GPU’s advantage here did not come from any single operation being fast. It came from doing a vast number of similar operations at the same time. That parallelism is the entire value proposition. If your code cannot express its work as “do all of this at the same time,” there is nothing for the parallelism to grab onto — and a GPU running serial, one-at-a-time work is just an expensive, awkward, badly-shaped CPU.

So “put it on the GPU” was never the real instruction. The real instruction was “reshape the work so it can be done all at once.” The hardware was only ever going to be as fast as the shape of the problem allowed.

The tool cannot save the wrong shape

The lesson generalizes well beyond GPUs. Reaching for faster hardware does little if the structure of your computation is fighting that hardware. For work like this, performance lived more in the shape of the computation — how it was organized, batched, and expressed — than in the nominal speed of the thing running it. You can put a slow algorithm on a fast machine and get a fast disappointment.

I keep that slowdown in mind as a reminder that “I made it faster” and “I moved it to faster hardware” are not the same sentence. The first is about understanding the work. The second is about hoping the silicon will understand it for you. It won’t.

— No signals, no returns, not investment advice.