Arithmetic Intensity: Why Most Kernels Never Touch the Peak

Two numbers describe a processor to a first approximation: how many floating-point operations it can do per second, and how many bytes per second it can move to and from memory. An operation's speed is decided by which of those it exhausts first, and the quantity that decides it is the ratio of the two:

arithmetic intensity = FLOPs performed / bytes moved

Compare that ratio to the machine's own FLOPs-per-byte and you know, before running anything, which ceiling an operation works against — a bound, not a forecast: what cannot be beaten, not what an implementation will reach.

The crossover has a name — the ridge point — and it is not one number. It is peak compute over achievable bandwidth, and both terms move. Measured on a Tesla T4: 6.81 TFLOP/s of FP32 against a 275 GB/s read roof puts the ridge at 25 FLOPs per byte; a kernel that reads and writes gets only 229 GB/s, pushing its ridge to 30; and on the FP16 tensor cores, 20.8 TFLOP/s sustained moves it to 76 — or 146, if you count the 40.3 TFLOP/s the card hits in short bursts before its 70 W cap pulls it back. The same kernel can be compute-bound against one roof and memory-bound against another.

Below the ridge the arithmetic units idle waiting for data — memory-bound, speed is bandwidth × intensity, and peak compute is not the limit. Above it the memory system keeps up and the operation is compute-bound. Plot speed against intensity and you get the roofline: a diagonal rising with bandwidth, then a flat ceiling at peak compute.

Where ordinary neural-network operations fall is not close. Element-wise work — addition, ReLU, a multiply-add — sits far to the left of any ridge: a couple of FLOPs for every four or eight bytes read and written, an intensity of 0.08 to 0.25. Measured, those kernels reach 84–90% of the memory roof, so there is little left to win without moving fewer bytes. Fusion is the lever that does exactly that: it deletes the intermediate writes and reads between two kernels rather than trying to speed up arithmetic that was never the cost.

Not everything memory-bound is that tight. Softmax and layer norm sit at 65% and 63% of their roof — still memory-bound, but with real headroom, because each makes more than one pass over the data. A plain sum, a single pass, hits 98%. "Memory-bound" names the ceiling that applies, not a promise that you are touching it.

Matrix multiplication is the exception that pays for the whole machine. A well-tiled multiplication of two n × n matrices does 2n³ FLOPs while ideally moving only O(n²) values to and from memory, so reuse — and intensity — grows with n. That is the entire reason deep learning runs on GPUs.

The word doing the work is ideally. Measured here, a 256³ matmul has intensity 85 and is still memory-bound at 4% of its roof; 2048³ touches the roof exactly; 4096³ and 8192³ fall back to 42% and 50%. High intensity buys the right to be compute-bound — whether an implementation collects is a separate question. And a matmul with one small dimension, like a single-row activation in LLM decoding, falls off the roof entirely and behaves like an element-wise kernel.

The measured corner is softer than the model. The real turn happened between 16 and 24 FLOPs per byte rather than exactly at 30 — caches, launch overhead and imperfect overlap round it off. The roofline is a bound and a diagnosis, not a prediction to three digits.

It also explains GPU pricing. Compare an H100 and an H200: the same compute, 43% more memory bandwidth and 76% more memory, and the H200 sold as the better card for serving language models. That only makes sense once you know LLM decoding is memory-bound — bandwidth is the performance number, and the extra capacity buys the bigger model, longer KV cache and larger batch that turn it into throughput.

The practical habit: before optimizing a kernel, compute its intensity and compare it to the ridge for the roofs you are actually on. Far below, optimizing arithmetic alone is unlikely to help — reduce bytes moved, fuse operations, change precision or layout. Above, the arithmetic is the target.

Worked out on a real T4, with the measured roofline and where each layer type lands on it, in How GPUs Actually Run Machine Learning. The decoding case is in How LLM Inference Actually Works.