Batch normalisation: keeping each layer on a stable footing
Before each layer of a network does its work, measure the average and the spread of what just arrived across a small batch of examples, and rescale it to a standard size. Then let the layer stretch and shift it back if it wants to. With that step, and the bolder training it allowed, a 2015 image network reached the same accuracy in a fourteenth of the training steps.
A standardising step inside the network
A deep network is trained in small batches: it looks at a few dozen examples, nudges every weight, and moves on. Batch normalisation, proposed by Sergey Ioffe and Christian Szegedy, adds a step between layers. Take one feature — one number per example — across the batch. Subtract the batch's mean and divide by its standard deviation, so the batch now has mean zero and spread one. Then multiply by a learned number, called gamma, and add another, beta. Each feature gets its own gamma and beta, learned along with the ordinary weights.
The problem it was aimed at
A layer's inputs are the outputs of every layer below it. Each update to those lower layers changes what the upper layers receive, so each layer is learning against inputs that keep moving. The paper calls this internal covariate shift and says it forces small learning rates and careful choice of starting weights. It also points at a specific failure. A sigmoid unit squashes its input into the range 0 to 1, and its slope is close to zero once the input is far from zero. If drifting inputs push a unit out there, almost no gradient flows back through it and learning stalls. The deeper the network, the more the drift compounds.
Interactive Drag input mean and input spread to make the inputs drift, the way they do when lower layers change. Watch how much slope the sigmoid has left to pass gradient back. Then switch batch normalisation on, move gamma and beta, and press undo it to set them so the layer gets its original inputs back.
Why the undo step matters
Forcing every input to mean zero and spread one could cost the network something. The paper's example is that a sigmoid fed only standardised inputs is held to its nearly straight middle. The learned gamma and beta fix this. Set gamma to the batch's original spread and beta to its original mean and the layer gets its old inputs back. So the step can always act as "do nothing" if that is what training finds best. The network decides where each feature sits, and the drift no longer decides it by accident.
Two shortcuts make it cheap. Each feature is standardised on its own, not decorrelated from every other feature, which would need a covariance matrix and its inverse square root at every step. And the statistics come from the current batch rather than the whole training set. In a convolutional layer, one feature map shares one mean, one spread, and one gamma and beta across every position in the image.
Training and use are different
During training, an example's normalised value depends on which other examples share its batch. The authors found that this jitter helps the network generalise, and that dropout could be removed or weakened. Once training is done, though, a prediction should depend only on its own input. So the batch statistics are replaced by fixed estimates for the whole population, taken from the training batches. The step then becomes one fixed scale and shift per feature.
What it bought
On an ImageNet classifier of the Inception family, adding the step alone matched the original's accuracy in under half the training steps. With a learning rate five times higher, plus other changes it made possible, the same accuracy took 14 times fewer steps. The same raised learning rate without it drove the original network's weights to overflow. A sigmoid version of the faster setup reached 69.8% accuracy. The original network with sigmoids never did better than chance, the authors report. An ensemble of six batch-normalised networks reached 4.9% top-5 error on the validation set, which the authors note beats the estimated accuracy of human raters.
Why it works is still argued
The method has outlasted its original explanation. In 2018 Shibani Santurkar and colleagues trained networks with batch normalisation and then injected noise right after each normalising step, with a non-zero mean and a spread that changed at every step. That deliberately restored the shifting inputs. The noisy networks trained almost exactly as well as the clean ones, and both clearly beat networks with no normalisation. Their reading is that batch normalisation helps mainly by making the loss surface smoother, so a gradient step goes about where the gradient says it will, and larger steps are safe. The original paper had itself left the effect on gradient flow as an open question.
In short
Standardise each feature using the batch's own statistics, then let a learned scale and shift put it wherever it is most useful. Training tolerates far higher learning rates and less careful starting weights, gets some regularisation for free, and finishes in a fraction of the steps. The step was designed to stop each layer's inputs from drifting. Later work suggests it helps more by making the loss surface smoother.
Where this comes from
- Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift linked only, not reproduced
arxiv.org/abs/1502.03167 - How Does Batch Normalization Help Optimization? linked only, not reproduced
arxiv.org/abs/1805.11604