Sequence-Level vs Token-Level Distillation

"We distilled it from a bigger model" covers at least three different procedures, and they differ in what they need from the teacher and in how much signal they carry per example.

token-level     match the teacher's full next-token distribution
                at every position            → needs logits (weights, or an API that returns them)

top-k           the same, truncated to the k most likely tokens
                                             → what some APIs expose as top-k log-probabilities

sequence-level  let the teacher write, train on its text as ordinary data
                                             → needs only outputs

Training on the teacher's text is still distillation, in a much noisier form. Text sampled from the teacher is a Monte Carlo estimate of the sequence-level objective: instead of the full distribution at each position, the student sees one draw from it. Nothing is lost in principle — drawn from the teacher's actual distribution, those samples estimate the objective in expectation — but one outcome carries far less than a probability vector, so it takes far more data to get the same signal. The word doing the work is actual: temperature, top-p or greedy decoding sample a different distribution, and the estimate is then of something else.

The gap shows up most clearly when data is scarce. At a fixed budget of 250,000 training characters in a small controlled setup, the student reached 2.50 bits per character from full distributions, 4.26 from the same amount of real text (no teacher at all), and 5.27 from sampled teacher text. At 2 million characters the gap narrows sharply — 2.02, 2.08 and 2.33 — but does not close: 2.33 against 2.02 is about 24% more perplexity, well outside the spread between seeds, while the 2.02-against-2.08 ordering is not. One caveat the comparison cannot remove: the sequence students train on text the teacher wrote and are tested on real text, so richer supervision and a shifted training distribution change together.

Here top-k cost something quieter than accuracy: variety. Truncating to the five most likely tokens never shows the student the tail of the teacher's distribution, and in this run the student wrote text with about a quarter fewer distinct four-character sequences than the others, at both budgets. Its loss looked fine; its outputs were narrower.

Greedy continuations are a different distribution entirely. Training on the teacher's single most likely continuation does not sample its distribution at all — it imitates a narrow, deterministic caricature of it. In the same setup that student reached 4.33 bits per character, the worst of the five routes, and produced text with barely more than a quarter of the teacher's variety, looping through the teacher's favourite phrases.

That verdict belongs to open-ended generation, where the collapse is the whole problem. Where the right answer is largely determined — translation is the standard case — that same collapse makes the target easier for a small model to fit, and training on the teacher's most likely output is an established way to distil. Sample when the student has to learn a distribution; decode deterministically when the task has one intended answer.

And the two routes cost very differently. Reading out the teacher's distributions over 2 million characters of existing text took 4 seconds. Having the teacher write 2 million characters took over five minutes — generation is sequential, scoring is not. At this scale, with a 96-character vocabulary, token-level is both the cheaper and the richer option. At LLM scale the expensive part moves: full distributions over a 100,000-token vocabulary run to hundreds of gigabytes for a few million tokens, which is why top-k logits and sequence-level routes exist at all.

Two practical notes. Any student trained on fixed data is scored on contexts it did not produce and must then continue from its own early mistakes — a mismatch that affects both routes, not only the sequence one. On-policy variants address it directly: the student generates, and the teacher scores its predictions on the student's own contexts. And using a commercial model's outputs to train a competing model is frequently forbidden by its terms of service, whatever the technical merits.

Related: SFT is the same mechanics with human-written targets, and RLAIF transfers a teacher's judgements rather than its text. The measurements above, at the smallest scale where the differences become visible, are in Knowledge Distillation: What a Teacher Actually Teaches.