№ 120 · machine learning

Adam: giving every weight its own step size

Training a neural network means nudging millions of weights downhill on an error surface, one small step at a time. Adam is a rule for sizing those steps, and it gives every weight its own. It keeps two running averages for each weight, one of its recent slopes and one of their squares, and divides the first by the square root of the second.

One step size is the wrong size for most weights

Plain gradient descent measures the slope of the error with respect to each weight and moves every weight against its slope, scaled by a single number, the learning rate. That works when the surface slopes about equally in every direction. It rarely does. Picture a long, narrow valley: steep walls across it, a gentle floor along it. A learning rate small enough not to bounce off the walls is far too small to make progress along the floor, and one large enough for the floor sends the walls into ever-wider swings. In a real network the gradients can differ hugely from one layer to the next, and in stochastic training each slope is measured on a small random batch of examples, so it is noisy as well.

The rule: average the slope, divide by its typical size

Diederik Kingma and Jimmy Ba proposed Adam in 2014, naming it for adaptive moment estimation. For every weight it keeps two numbers. The first is a running average of that weight's recent gradients, which smooths out batch noise much as momentum does. The second is a running average of the squared gradients, a measure of how large that weight's gradients usually are. Each step moves the weight by a base step size, α, times the first average divided by the square root of the second. A tiny constant, ε, is added under the division so it never divides by zero.

Interactive Top: a long, narrow valley, ten times narrower than it is long, and two optimisers starting from the same point. Drag steps to scrub through training, and drag each step-size slider to find where plain gradient descent starts to bounce off the walls while Adam still walks the floor. noise makes every measured slope noisy, as a small batch would. Bottom: the counterexample of Reddi, Kale and Kumar. Drag α and watch where Adam and its repair, AMSGrad, settle.


Top: the valley is the error surface ½(x² + 100y²), so it is 10 times narrower than it is long and its walls curve 100 times more sharply than its floor; the vertical axis is stretched to fit. Both optimisers start at (−4.5, 0.9) and aim for (0, 0). Plain gradient descent moves each coordinate by the learning rate times its slope; on this surface it diverges for any rate above 0.02, since the wall then sends each step back past the bottom by more than it came. Adam uses the paper's default averaging rates, 0.9 and 0.999, and ε = 10−8. With noise on, each slope gets random error, ten times larger across the walls than along the floor, from a seeded generator, so a given new noise setting is reproducible. Bottom: the synthetic problem in Reddi, Kale & Kumar's experiments. One weight lives on [−1, 1] and starts at 0; on every 101st step its gradient is 1010, on the other 100 steps it is −10. Over each cycle the pushes add to +10, so the best place is −1. Both methods use averaging rates 0.9 and 0.99, the setting they report, and here both include Adam's start-up correction. The paper's text gives no step size for this run, so α is a constant of our choosing. The readout averages each method's position over its last 101 steps, one full cycle. Everything is computed in your browser; nothing is the papers' data.

The division is what makes it work. If a weight's gradients have been large, the denominator is large, and the step is reined in. If they have been small, the step is scaled up. When the gradients point the same way step after step, the ratio is close to one and the weight moves by about α. Kingma and Ba show the step is roughly bounded by α in ordinary conditions, so α says how far any weight may move in one step, a little like a trust region. Multiplying every gradient by the same constant changes nothing, apart from the tiny ε, since it multiplies the top by that constant and the bottom by the square root of its square. And when the gradients start to disagree, as they do near a minimum where noise dominates the signal, the average shrinks against its typical size and the steps shrink on their own.

A correction for starting at zero

Both averages start at zero, so for the first steps they are pulled towards zero. The authors divide each by a factor that undoes that pull and fades to nothing as training goes on. This matters most when the average of squares is set to remember a long history, which the authors recommend for sparse gradients. The uncorrected denominator then starts far too small and the first steps come out much too large. They tested this by training a variational autoencoder with and without the correction: without it, settings that average over a long history became unstable, especially in the first epochs. Their suggested defaults are α = 0.001, averaging rates of 0.9 for the gradient and 0.999 for its square, and ε = 10−8.

Where it came from and what it showed

Adam joins two earlier methods. AdaGrad divides by the root of all squared gradients so far, which suits rare, sparse features but makes steps shrink steadily over a long run. RMSProp uses a running average of the squares instead, which can follow a target that changes. Adam takes the running average and adds the smoothed gradient on top and the start-up correction. The cost is two extra stored numbers per weight. The paper tested it on logistic regression for handwritten digits, where it kept pace with gradient descent with momentum and both beat AdaGrad; on sparse word counts from movie reviews, where it was as fast as AdaGrad; and on networks with and without dropout, where it did better than the other methods tried. On a convolutional network it was only slightly better than gradient descent with momentum, but it set a sensible step for each layer without one being picked by hand.

The catch: the proof had a hole

The paper includes a guarantee that, on convex problems, Adam's average regret goes to zero. In 2018 Sashank Reddi, Satyen Kale and Sanjiv Kumar showed the proof relies on an assumption that does not hold. They built a one-weight counterexample on the interval from −1 to 1: on one step in every three the gradient is a large positive number, and on the other two it is −1. The large gradients win on balance, so the best place to be is −1. In their setting Adam heads to +1, the worst point available. The large gradient arrives rarely, inflates the average of squares that divides it, and that memory fades before the next one comes, so the frequent small pushes the wrong way win. Their fix, AMSGrad, divides by the largest average of squares seen so far, so the scale of a weight's steps can never grow back after a big gradient.

In short

Adam keeps, for every weight, a smoothed gradient and a smoothed squared gradient, and steps by α times the first over the root of the second. Weights with big gradients take smaller steps, weights with small ones take bigger steps, and noise near a minimum shrinks the steps automatically. A correction keeps the first steps honest. It is not guaranteed to converge, and a constructed problem can send it to the worst point available.

Where this comes from

  1. Adam: A Method for Stochastic Optimization linked only, not reproduced
    Diederik P. Kingma and Jimmy Ba · arXiv:1412.6980 · 2014
    arxiv.org/abs/1412.6980
  2. On the Convergence of Adam and Beyond linked only, not reproduced
    Sashank J. Reddi, Satyen Kale, Sanjiv Kumar · arXiv:1904.09237 · 2019
    arxiv.org/abs/1904.09237