Why a small model learns more from a big model's wrong answers

A large network's mistakes are not noise. The tiny probability it gives each wrong answer maps what it has learned — and a small network can be trained on that map instead of the labels.

Two kinds of target

A classifier is normally trained on hard targets: this picture is a 2, all else zero. A trained network never quite says that. It says 2 with probability near one and hands tiny leftovers to 3, 7, 8. Those leftovers are the soft targets. Distillation trains a small model to reproduce a big model's soft targets instead of the labels.

Why bother

The best classifier is often an ensemble — many networks averaged — or one very large network trained slowly under heavy regularisation. Both are too expensive to serve millions. Hinton, Vinyals and Dean, building on Caruana and collaborators' work on compressing ensembles, showed you can train something cheap to generalise the way the expensive thing does. On MNIST, their large net (two layers of 1200 units, dropout, jittered inputs) made 67 test errors; a plain small net of two 800-unit layers made 146. The same small net, given the added task of matching the large net's soft targets at T = 20, made 74. In a speech-recognition acoustic model, more than 80% of a ten-model ensemble's frame-accuracy gain carried over to a single distilled net.

Where the knowledge hides

The paper's example: one handwritten 2 might get probability 10⁻⁶ of being a 3 and 10⁻⁹ of being a 7; another 2 might have them the other way round. Those ratios say which 2s look like 3s and which look like 7s — a similarity structure the hard label throws away. Or their other image: a BMW has a very small chance of being mistaken for a garbage truck, but that mistake is still many times more probable than a carrot.

But 10⁻⁶ is invisible to training. The usual loss, cross-entropy, barely moves when so small a probability changes. So the paper introduces a temperature. A softmax turns raw scores, the logits z_i, into probabilities proportional to exp(z_i/T). Normally T = 1. Raising T divides every logit before exponentiating, shrinking the gaps and flattening the distribution until the tiny probabilities are visible. Teacher outputs are computed at high T; the student matches them at the same T, then runs at T = 1.

With true labels available, the paper adds a smaller second term: cross-entropy against the hard label at T = 1. The gradients from the soft targets scale as 1/T², so the paper multiplies them by T² to keep the two terms in balance when T changes. In the high-temperature limit, with logits centred at zero, matching soft targets becomes minimising the squared difference of logits — Caruana's method is a special case.

How much temperature

Not as much as possible. With 300 or more units per layer, every temperature above 8 gave similar results; with the student cut to 30 units per layer, 2.5 to 4 worked significantly better. The paper's reading: a tiny student benefits from ignoring the teacher's most negative logits, which are poorly constrained and possibly noisy.

Starkest: they removed every 3 from the transfer set. The student made 206 test errors, 133 on the 1010 test threes; after raising the bias of the 3 class by 3.5, it made 109 errors and got 98.6% of the threes right — having never seen one. All it knew about 3s came from how the teacher spread probability across the other digits.

In short

Distillation raises the softmax temperature until a network's wrong-answer probabilities are large enough to train on, then teaches a small network to reproduce them. The student inherits the teacher's judgement about what resembles what, which no bare label carries.