Born-Again Networks: A Student With No Teacher to Compress

Distillation is introduced as compression: a big teacher, a small student. Remove the compression and the procedure still runs. Train a network on labels, then train a second network of exactly the same architecture on the first one's probabilities, then a third on the second's. That is a born-again network, and the interesting part is that the copy is sometimes better than the original.

That should be impossible if distillation were copying. A student trained to reproduce another model's outputs, with no extra capacity and no extra data, ought to land at best where its teacher did. It does not, and the reason is that the target has changed: a trained network's probabilities are not its training labels. They are a smoothed, per-example version of them — the model's own opinion about which examples are ambiguous.

When it works, and when it does not, comes down to memorization. In one controlled run, generations trained on 5,000 images went 67.0% → 68.5% → 68.7% → 68.9%. On all 50,000 the same procedure ran the other way: 82.8% → 82.1% → 81.5% → 81.0%.

Read those two rows against the spread between seeds, about half a point, and they stop being mirror images. Each of the three students does beat the particular model that taught it — but almost the whole climb is the first step, +1.5 points, and the two after it add 0.1 and 0.2, inside the noise. The losses are the opposite: 0.7, 0.6, 0.6, every one of them outside it. One generation buys the gain and the next two hold it; in the other regime, every generation costs.

The training accuracies explain the split. On 5,000 images, generation 0 fits 100% of its own training data while getting a third of the test set wrong — it has memorized. Its probabilities are that memorization, smoothed, and a smoothed version of a memorized dataset turns out to be a better target than the labels it came from. On 50,000 images there is much less to launder: generation 0 fits 92.9% of its training data, each generation after it fits less (87.2%, 85.8%, 85.0%) and tests worse with it. Past the point where smoothing helps, more smoothing is just a weaker signal.

And note what the winning students are not doing: ducking the target. Every generation on 5,000 images also reaches 100% training accuracy — it fits the teacher's answers as completely as the teacher fitted the labels. The gain cannot be that the student failed to copy something harmful. It is that matching a distribution is a different job from matching a label: the same 100% hides a target that says which examples were hard and which classes they were confused with, and fitting that lands the network somewhere else.

There is a second, cruder way to beat a teacher, and it is worth telling apart. Take labels where one in five was replaced by a random class, and a teacher pushed hard enough to memorize 90% of those wrong labels; it falls to 72.9% test accuracy. A student with a tenth of its parameters, distilled from it, reaches 75.5% — better than the model it copied — because a network that small cannot reproduce a memorized list of arbitrary answers and averages them away instead.

That one really is about capacity, and it needs the size gap that born-again networks do not have. Two different routes to the same surprise: the student is too small to inherit the damage, or the target it inherits is better than the labels. Only the second survives when teacher and student are the same network.

So "the student can't beat the teacher" was never a conservation law. What a network ends up with is set jointly by the target function, its own capacity and the optimization — and distillation changes the first of those, which is enough on its own.

Practically: born-again training is a regularizer that costs a full extra training run per generation, and it pays only when the original model is overfitting enough to be worth smoothing. The gap between training and test accuracy is the thing to check, and here it was decisive: 33 points on the 5,000-image run (100% against 67%) and the generations climbed; 10 points on the 50,000-image run (92.9% against 82.8%) and they drifted down. Budget one generation, not four.

Measured across four generations at two dataset sizes in Knowledge Distillation: What a Teacher Actually Teaches.