Prefill vs Decode: The Two Halves of an LLM Request

A language model answering a request does two things that look alike in code and behave nothing alike on the chip.

prefill   read the whole prompt          — one pass, many rows at once
decode    write one token, then another  — one pass per token, one row each

Same function, same weights, different arithmetic intensity — and that single fact explains most of what a serving system does.

Prefill is the workload a GPU likes. A 1,024-token prompt goes through each weight matrix as a 1,024-row matrix multiplication: every weight fetched from memory is used a thousand times. Measured on a T4, the prefill span for that prompt was 137 ms, which for the 1.06 TFLOP of arithmetic in it is 7.7 TFLOP/s. The same card sustains 20.8 TFLOP/s on a pure matmul benchmark, and the best any prefill reached here was 9.1 — a real model is never a clean matmul. Not at the roof, but on the right wall.

Decode is not a computation, it is a flow. Each step after the first feeds in a single row. The weights read from memory are the same; the work done with them shrinks a thousandfold. Matrix–matrix becomes matrix–vector, and the textbook conclusion follows: a step should cost what it takes to stream the model past the arithmetic units, once per token.

time per token ≳ model size in bytes / effective memory bandwidth

For a 0.5B model in half precision — 0.92 GiB of weights, on a card measured at 274 GB/s — that is 3.7 ms.

The measured step was 33 ms — nine times the floor, and the gap is not bandwidth. The step moved bytes at 31 GB/s, eleven percent of what the same card had been measured delivering minutes earlier. Under a profiler the GPU was busy 10.4 ms of a 40 ms step, assembled from 1,415 separate kernel launches: most of a decode step is the gaps between them. Capture the same arithmetic into a CUDA graph, so launches are replayed instead of issued, and the step falls from 39.0 ms to 13.1 ms — three times faster without changing a byte of the computation.

The formula is a floor, not a forecast — what decode converges to once enough work is in flight to hide the cost of starting it. At batch 1 on a small model you are nowhere near it, and the thing to fix is not the memory system.

The user sees the split as two waits. Time to first token covers everything up to the first character: queue, tokenization, prefill. Time per output token is one decode step — 33 ms here, about 30 a second. TTFT grows with the prompt and depends on the attention implementation; TPOT barely moves with it, going 33.5 → 34.4 ms across a 64-fold increase in context, 128 tokens to 8,192.

That flatness expires. Each step also reads the KV cache, which grows with context while the weights do not — but at 8,192 tokens it is 96 MiB against 942 MiB of weights, and the step is launch-dominated anyway. Push context or batch far enough and KV traffic is the term that matters: "context only moves TTFT" is a rule of this regime, not of the arithmetic.

Short prompts are nearly free, for the reason decode is slow. A 16-token prefill took 36.4 ms at 0.43 TFLOP/s, and 256 tokens took 37.0 ms: a pass that small is not worth starting, so the time is overhead rather than arithmetic — the same overhead, in the same place, as the gaps in a decode step. Prompt length starts costing once prefill saturates the machine, between 256 and 512 tokens here.

Batching helps decode most, and prefill more than the slogan admits. A second sequence rides the same weight stream almost for free: from batch 1 to 16 the step stayed near 33 ms while throughput rose from 30 to 445 tokens a second, and only at 32 did per-sequence latency start paying — 55 ms a step for 582 a second in total. A long prompt carries parallelism of its own and gains less; a short one, at two percent of the card's throughput, has plenty left to amortize.

Practically: if first-token latency is the problem, look at prompt size and the attention implementation; if throughput is the problem, look at batch size. And before deciding decode is bandwidth-bound and shopping for a card with more bandwidth, measure what you are actually achieving — at a tenth of the sticker, the next win is in the runtime.

Measured end to end on a real GPU, with the traces, in How LLM Inference Actually Works. The hardware reasoning behind "memory-bound" is in How GPUs Actually Run Machine Learning.