Why temperature=0 Is Not Bit-Exactly Deterministic

Set temperature=0 and the sampling step stops being random: as T falls to zero the softmax concentrates all its mass on the highest-logit token, so the draw is an argmax and that token is always chosen. Every guide says this makes generation deterministic, and for a single process on a fixed machine it very nearly is.

Across a real serving stack it is not, and the reasons have nothing to do with sampling.

Floating-point addition is not associative. (a + b) + c and a + (b + c) can differ in the last bits. A GPU reduction — summing thousands of products to get one logit — splits that sum across threads in whatever shape the kernel chose, and the shape can change with batch size, with a library version, or with an autotuner's decision. The logits come out equal to within a rounding error, which is harmless everywhere except at one place: when two candidate tokens are nearly tied, the rounding decides which one is the maximum. One flipped token, and the continuation diverges from there.

You are sharing a batch with strangers. A serving system batches whatever requests happen to arrive together, and the batch is not an inert container: its shape selects which kernels run and how each reduction is split and ordered. Change the batch and the arithmetic changes, so the same prompt can come back with logits that differ in the last bits depending on how busy the server was. The property that would prevent this — producing the identical result for a request regardless of what else is in the batch — is batch invariance, and ordinary high-performance kernels do not have it.

Mixture-of-experts routing can add batch-dependent effects of its own, since some implementations balance load or cap capacity across the tokens in a batch. But that is an extra source, not the cause: dense transformers with no routing anywhere show the same behaviour.

Both are amplified by autoregression. A single flipped token at position 12 changes the context for position 13, and the answers separate from there. This is why "usually identical, occasionally completely different" is the observed pattern, rather than small uniform noise.

What follows for engineering practice:

• Do not assert on exact strings in tests against a hosted model you do not control. Assert structurally: valid JSON, a schema that parses, the required fields present, a number within tolerance, a classification label out of a known set.
• Pin what you can — model version, prompt, decoding parameters — and accept that the remaining variance may be coming from the serving stack rather than from your own logic.
• If you genuinely need bitwise reproducibility, you have to control the whole stack: weights, numerical precision, kernels, decoding rules, execution environment. Batch-invariant kernel implementations exist and can hold determinism even under dynamic batching, at a throughput cost; running single-tenant with a fixed batch is the blunter way to get there. Either way it is a deployment decision, not a parameter you can send in a request.
• Cache instead of hoping. For workloads where the same input must give the same output — evaluation baselines, golden files, anything a user might compare against — store the first answer and reuse it rather than regenerating and diffing. Note what that buys: a stable answer, not a correct one, so the cached value still needs to have been checked once.

The useful mental correction: temperature=0 removes sampling randomness. It does not remove the numerical and scheduling nondeterminism underneath, and those live in a layer you usually do not control.

The decoding knobs and the rest of the practical API surface are in How LLM Generation Works; what the batch is doing to your request on the GPU is in How LLM Inference Actually Works. Related short: temperature, top_k, top_p.