№ 118 · machine learning

Batch normalisation: keeping each layer on a stable footing

Before each layer of a network does its work, measure the average and the spread of what just arrived across a small batch of examples, and rescale it to a standard size. Then let the layer stretch and shift it back if it wants to. With that step, and the bolder training it allowed, a 2015 image network reached the same accuracy in a fourteenth of the training steps.

A standardising step inside the network

A deep network is trained in small batches: it looks at a few dozen examples, nudges every weight, and moves on. Batch normalisation, proposed by Sergey Ioffe and Christian Szegedy, adds a step between layers. Take one feature — one number per example — across the batch. Subtract the batch's mean and divide by its standard deviation, so the batch now has mean zero and spread one. Then multiply by a learned number, called gamma, and add another, beta. Each feature gets its own gamma and beta, learned along with the ordinary weights.

The problem it was aimed at

A layer's inputs are the outputs of every layer below it. Each update to those lower layers changes what the upper layers receive, so each layer is learning against inputs that keep moving. The paper calls this internal covariate shift and says it forces small learning rates and careful choice of starting weights. It also points at a specific failure. A sigmoid unit squashes its input into the range 0 to 1, and its slope is close to zero once the input is far from zero. If drifting inputs push a unit out there, almost no gradient flows back through it and learning stalls. The deeper the network, the more the drift compounds.

Interactive Drag input mean and input spread to make the inputs drift, the way they do when lower layers change. Watch how much slope the sigmoid has left to pass gradient back. Then switch batch normalisation on, move gamma and beta, and press undo it to set them so the layer gets its original inputs back.

One feature of one layer, feeding a sigmoid. Each batch is built from fixed standard-normal draws (a seeded generator, so a given batch is reproducible), scaled by input spread and shifted by input mean. With batch normalisation on, the batch's own mean and standard deviation are subtracted and divided out (with ε = 10−5 added to the variance, as in the paper), then the result is multiplied by gamma and has beta added. The slope readout is the sigmoid's derivative g(x)(1 − g(x)) averaged over the batch: it is what scales the gradient passed back through the unit, and its peak is 0.25 at an input of zero. Undo it sets gamma and beta to this batch's measured spread and mean; the readout then shows how far the outputs are from the raw inputs. A small batch's measured mean and spread wander from the values set on the sliders, which is the batch noise the paper credits with some regularisation. Nothing here is trained; the numbers are illustrative, not from the paper.

Why the undo step matters

Forcing every input to mean zero and spread one could cost the network something. The paper's example is that a sigmoid fed only standardised inputs is held to its nearly straight middle. The learned gamma and beta fix this. Set gamma to the batch's original spread and beta to its original mean and the layer gets its old inputs back. So the step can always act as "do nothing" if that is what training finds best. The network decides where each feature sits, and the drift no longer decides it by accident.

Two shortcuts make it cheap. Each feature is standardised on its own, not decorrelated from every other feature, which would need a covariance matrix and its inverse square root at every step. And the statistics come from the current batch rather than the whole training set. In a convolutional layer, one feature map shares one mean, one spread, and one gamma and beta across every position in the image.

Training and use are different

During training, an example's normalised value depends on which other examples share its batch. The authors found that this jitter helps the network generalise, and that dropout could be removed or weakened. Once training is done, though, a prediction should depend only on its own input. So the batch statistics are replaced by fixed estimates for the whole population, taken from the training batches. The step then becomes one fixed scale and shift per feature.

What it bought

On an ImageNet classifier of the Inception family, adding the step alone matched the original's accuracy in under half the training steps. With a learning rate five times higher, plus other changes it made possible, the same accuracy took 14 times fewer steps. The same raised learning rate without it drove the original network's weights to overflow. A sigmoid version of the faster setup reached 69.8% accuracy. The original network with sigmoids never did better than chance, the authors report. An ensemble of six batch-normalised networks reached 4.9% top-5 error on the validation set, which the authors note beats the estimated accuracy of human raters.

Why it works is still argued

The method has outlasted its original explanation. In 2018 Shibani Santurkar and colleagues trained networks with batch normalisation and then injected noise right after each normalising step, with a non-zero mean and a spread that changed at every step. That deliberately restored the shifting inputs. The noisy networks trained almost exactly as well as the clean ones, and both clearly beat networks with no normalisation. Their reading is that batch normalisation helps mainly by making the loss surface smoother, so a gradient step goes about where the gradient says it will, and larger steps are safe. The original paper had itself left the effect on gradient flow as an open question.

In short

Standardise each feature using the batch's own statistics, then let a learned scale and shift put it wherever it is most useful. Training tolerates far higher learning rates and less careful starting weights, gets some regularisation for free, and finishes in a fraction of the steps. The step was designed to stop each layer's inputs from drifting. Later work suggests it helps more by making the loss surface smoother.

Where this comes from

  1. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift linked only, not reproduced
    Sergey Ioffe, Christian Szegedy · arXiv:1502.03167 · 2015
    arxiv.org/abs/1502.03167
  2. How Does Batch Normalization Help Optimization? linked only, not reproduced
    Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, Aleksander Madry · arXiv:1805.11604 · 2018
    arxiv.org/abs/1805.11604