Adversarial examples: fooling a network with a change too small to see
A trained image classifier can be fooled by a picture that looks, to you, exactly like one it gets right. Every pixel is moved by an amount too small to see, but each in a chosen direction, and the network changes its answer, often with high confidence. These pictures are called adversarial examples. The surprising part is the cause: not that networks are too strange, but that they are too nearly linear.
A change too small to see
In 2013 Christian Szegedy and six colleagues reported that they could take a test image a network classified correctly and search for a small change to it that maximised the network's error. The change they found was hardly perceptible, yet the network now gave a wrong answer. They named the results adversarial examples. They did not stop at one network. The same altered images were also hard for other networks with different numbers of layers or different settings, and even for a network trained on a different half of the handwritten-digit training set. A quirk of one training run would not travel like that. Something shared was going on.
The explanation: too linear, not too strange
Early guesses blamed the networks' extreme nonlinearity, or overfitting. In 2014 Ian Goodfellow, Jonathon Shlens and Christian Szegedy argued for the opposite. Start with the simplest model: a score made by multiplying each input by a weight and adding the results up. Now nudge every input by the same tiny amount, ε, up if its weight is positive and down if it is negative. Each term of the sum rises by ε times the size of its weight, and every one of those rises has the same sign, so none cancel. The score moves by ε times the sum of the weight sizes, which is ε times the number of inputs times the average weight size.
The nudge to any single input stays at ε however many inputs there are. The change in the score grows with their number. A 32 × 32 picture has 1,024 inputs; with weights averaging 0.5 in size, a nudge of ε = 0.01, about two and a half of the 255 grey levels an 8-bit image can store, moves the score by about 5. That is enough to carry a confident answer across the line to the other one. Many changes too small to matter one at a time add up, because all of them were chosen to point the same way. The authors point out that an 8-bit image already throws away anything finer than 1/255 of its range, so a sensible classifier ought not to care about a change of that size at all.
Interactive A made-up linear classifier looks at a grey picture and adds up weight × pixel. Above zero it says A, below zero B; the clean picture scores +2. Drag nudge ε to set how far each pixel may move, and picture size to change how many pixels there are. Switch between the aligned nudge, where every pixel moves against its weight, and a random nudge of the same size; press new random nudge to draw another.
Direction matters, not the exact point
The same size of nudge with random signs does almost nothing. Its terms push both ways and mostly cancel, so on average the score does not move at all, and a typical random nudge moves it far less than the aligned one. The paper's own control makes the point. Maxout networks trained on inputs with random noise of size ε added, instead of aligned nudges, were still fooled by aligned nudges 86.2% and 90.4% of the time, for the two kinds of noise they tried.
This also explains why the images travel between networks. What matters is the direction of the nudge, and the authors found that adversarial examples fill broad stretches along that direction rather than sitting in tiny isolated pockets. Their proposed reason the fooled networks agree is that networks trained on the same task end up resembling the same simple linear classifier, so a direction that misleads one tends to mislead the others in the same way.
A fast attack falls out of it
Deep networks are not linear, but they are built to behave nearly so. Rectified linear units, LSTMs and maxout networks are designed that way because it makes them easy to train. So the authors treated the network as if it were linear near the input: work out which way each pixel should move to raise the training error, using the same backpropagation that trains the network, and move every pixel ε in that direction. They called this the fast gradient sign method. On handwritten digits with pixel values from 0 to 1 and ε = 0.25, a shallow softmax classifier was wrong on 99.9% of the altered test digits, with an average confidence of 79.3%, and a deeper maxout network was wrong on 89.4%, with 97.6%. That ε is a quarter of the range and so is visible here, but these digits are close to black and white, and people still read them without trouble.
Training against it helps, but does not cure it
A cheap attack makes a cheap defence: mix freshly made adversarial examples into training, made against the network as it is at that step. On handwritten digits this lowered a maxout network's ordinary test error from 0.94% to 0.84%, and a larger network trained this way averaged 0.782%. Its error on fast-gradient-sign examples fell from 89.4% to 17.9%, but when it was fooled it was still 81.4% confident on average. An ensemble of twelve networks was fooled 91.1% of the time by examples aimed at the whole ensemble. A shallow network of radial basis function units, which respond only near stored points, was wrong 55.4% of the time but only 1.2% confident when wrong: it knew it was lost. The authors' conclusion is that ease of training and resistance to these nudges pull against each other. The models that are easy to optimise are the ones that are easy to perturb.
In short
An adversarial example is an input changed by a tiny amount in a chosen direction so that a model answers wrongly. In many dimensions, a tiny aligned change to every input adds up to a large change in a nearly linear score, while random noise of the same size mostly cancels. That makes the attack cheap, explains why one network's adversarial examples often fool another, and suggests why defending against them is hard.
Where this comes from
- Explaining and Harnessing Adversarial Examples linked only, not reproduced
arxiv.org/abs/1412.6572 - Intriguing properties of neural networks reuse permitted with attribution
arxiv.org/abs/1312.6199