Fidelity Is Not Quality: Two Ways to Grade a Distilled Model

There are two different questions to ask about a distilled model, and they are routinely collapsed into one.

fidelity  — how closely does the student reproduce the teacher?
            agreement on top-1, KL between their distributions,
            and how often it repeats the teacher's own mistakes

quality   — how often is the student right?
            accuracy against real labels

With an accurate teacher the two are forced close, and that is arithmetic rather than a finding. Agreeing with the teacher and being right are the same event everywhere except on the images the teacher gets wrong, so the two numbers cannot differ by more than its error rate: a 94% teacher pins them within six points whatever the student does. Which is why a pooled correlation flatters them. Across all 144 students in the lab agreement and accuracy correlate at 0.97, but that figure rides on the runs nobody would ship — among students above 70% accuracy, where a real candidate lives, it is 0.74.

More fidelity did not mean more quality. Distil a small network from an 83% teacher and you get 85.5% agreement and 77.8% accuracy. Distil the same network from a 94% teacher and agreement drops to 81.1% while accuracy rises to 80.1% — three seeds each, with no overlap between the two accuracy ranges. Teacher agreement on its own would have picked the worse model. And the two agreement numbers are not a common yardstick: each is measured against its own teacher, so the first student is closer to a worse function, not closer to the truth.

The sharpest form of the question is what happens where the teacher is wrong, and it is the one most often read wrong. The same student repeats the same wrong answer for 63.7% of the weak teacher's mistakes and 43.8% of the strong one's — which looks like a measure of inherited failure and is not. Two networks trained for the same task fail on many of the same images whether or not one taught the other, so the number needs a baseline: the same architecture trained on the labels, which never saw a teacher.

Against that baseline the cases separate completely. The strong teacher: baseline 42.5%, distilled 43.8% — 1.3 points, so almost none of it was inherited. The weak teacher: baseline 53.6%, distilled 63.7% — 10.1 points genuinely transferred. A teacher with a planted mistake: baseline 19.2%, distilled 72.8% — 53.5 points, nearly the whole number. One statistic, three meanings, and only the gap to the baseline tells them apart.

That last case is worth seeing in full. Take a teacher trained with every truck labelled automobile. Its student agrees with it 82% of the time and is right 72%. The same architecture trained on the true labels agrees only 70% and is right 78%. Higher fidelity, lower quality, and the gap is the flaw being faithfully reproduced.

So which do you want? It depends on what the distilled model is for.

• Replacing a model in production, where downstream systems are tuned to its behaviour: fidelity is what you are buying. A student that is more accurate but disagrees in 20% of cases is a migration, not a drop-in.
• Building the best small model you can afford: quality is the target and the teacher is scaffolding. A student that beats its teacher is a success, not an anomaly.
• Safety and audit: error agreement against its baseline is where inherited behaviour shows up — and it is half the question. A student also fails where its teacher does not, which no agreement number reports. The strong teacher's student is wrong on 19.9% of the test set; 15.4 of those points are images the teacher gets right, five and a half times the 2.8 points where the two are wrong together. Most of a student's failures are its own.

The practical rule is to report both, with their baselines. Agreement says how closely the student tracks its teacher — top-1 agreement most coarsely of all, since two models can pick the same class every time and still put very different probabilities behind it, which is what the KL divergence catches. Accuracy says whether the result is any good. A single "reaches 96% of teacher performance" hides which one was measured.

Worked through on 144 students, including the ones that copy faithfully and are worse for it, in Knowledge Distillation: What a Teacher Actually Teaches.