The Transfer Set: The Questions You Ask a Teacher

Distillation has an input nobody names in the loss function: the set of examples the teacher is run on. It is called the transfer set, and it is easy to treat as a formality — "the training data, obviously". It is not raw material. Every example in it is a question put to the teacher, the student learns the answers to those questions, and it must extrapolate everywhere else. In the measurements below — one student, one task, a fixed training budget — swapping the teacher for a better or worse one moved that student by 2.3 points. Swapping the questions moved it by tens.

It needs no labels. This is most of distillation's practical appeal. The targets come from the teacher, so unlabelled images, logged production inputs and scraped text are all fair game, and the set can be larger than the labelled one that trained the teacher. What it still needs is coverage: no labels does not mean no requirements, as the next two paragraphs measure.

What it covers is what transfers. A student distilled only on animal photos scored 0.08% on the vehicle test images — essentially nothing. The same architecture, the same teacher, asked about all ten classes instead: 87% on those same images. Knowledge is not a property of the teacher that flows across. It is reconstructed by the student from answers at specific points, and there were no points where vehicles lived.

Related data beats too little of the right data. The same teacher queried on 50,000 photos of a hundred other object classes produced a better student (67.9%) than 5,000 photos of the right classes (65.0%). Queries far off the task distribution stop helping, though: on pure random noise the student reached 13.1%, barely above guessing. A teacher asked about nothing teaches nothing.

More questions about the same images is a real lever. The widget's mixup of 5k bar is one version of this: 50,000 blends of two photos drawn from the same 5,000, asked about one by one, 70.6% against 65.0% for the 5,000 alone. Plain random crops of those same 5,000 do better still — 74.0%, +9.0 points over the uncropped 5,000, more than any change of teacher, loss or temperature in the same study. Every student here trains for the same 6,000 optimizer steps, so this is not extra training; it is the same budget spent on more places to ask about. No new picture was collected.

And it decides what can be cached. Because the teacher is frozen, its answers are usually computed once and looked up — far cheaper than running the teacher inside the training loop. With four-pixel random crops the cache holds up: putting the teacher back in the loop, so it answers about the cropped image rather than the original, bought 0.9 points for several times the training cost. It stops holding when the transformation makes a genuinely different picture. Interpolating the teacher's two stored answers for a blend, instead of asking it about the blend, lands on its actual top class only 86% of the time and costs 2.2 points. Cache when the approximation is good enough; re-query when the transformation changes what the teacher sees — and note that a harsher crop, one that can cut the object out, moves into the second category.

Choosing a fraction of it. If teacher passes are the budget, the intuition from active learning — spend them where the teacher is least certain — was the worst of the obvious options here: at a tenth of the data it scored 50.6% against 63.6% for a random tenth, 13 points worse, and the teacher's most confident tenth did best at 66.4%. The likely reason is that a student shown almost nothing but ambiguous cases never sees the typical members that establish what a class looks like — a hypothesis the experiment is consistent with rather than one it proves, since selecting by uncertainty also reshapes the class mix.

The short version: the teacher is a function, and distillation samples it. Which points you sample is part of the algorithm. The full set of measurements is in Knowledge Distillation: What a Teacher Actually Teaches.