How attention lets a model look back at everything

Attention lets a model look at every piece of its input at once and decide, for the job in front of it right now, how much each piece should count. Nothing is thrown away or memorised in bulk. Each step asks a question; the answer is a weighted mix of everything it can see.

The bottleneck it replaced

Early neural translators squeezed a whole sentence into one fixed-length vector, then generated the translation from that vector alone. Bahdanau, Cho and Bengio argued in 2014 that the single vector was a bottleneck: a long sentence is compressed just as hard as a short one, and earlier measurements showed the design losing quality quickly as sentences grew longer. Their fix: keep one vector per input word and let the decoder, at every output word, take its own weighted sum over all of them — a soft search for what matters now. Their model held up on 50-word sentences where the fixed-vector model fell apart, and its weights looked like an alignment: translating the man into l'homme, it read both English words to choose l'.

Queries, keys, values

Vaswani and colleagues in 2017 wrote the same idea in the form that has stuck. Every position offers three vectors: a key saying what it contains, a value it hands over if chosen, and a query — what it is looking for when it does the asking. The score between a query and a key is their dot product — multiply matching components, add them up — so a query pointing the same way as a key scores high; one at right angles scores zero.

The scores then go through a softmax: e to each score, divided by the total. This turns any list of numbers into positive weights summing to one, so the output is an honest average — every value times its weight, added up. Bahdanau's paper calls this an expected annotation: the mix you would get picking one source word at random with those odds. The softmax is also smooth — a slightly better score earns a slightly larger share — so it trains by gradient descent.

Why divide by √d

The full formula is softmax(QKᵀ/√d_k)V; the √d_k does real work. A dot product is a sum of d_k terms. If each query and key component is random with mean zero and variance one, the sum has variance d_k, so typical scores grow like √d_k. Push large scores through a softmax and one weight goes to nearly 1, the rest to nearly 0, and the slope goes nearly flat — the authors suspected this was why unscaled dot-product attention performed worse than the alternative at large d_k. Dividing by √d_k puts scores back on the scale the softmax handles well; their model used d_k = 64 per head.

What the weights do not tell you

An attention weight says how much of a value went into a mix. It is tempting to read them as the model's reasoning, but the Transformer paper only says attention could make models more interpretable and that some heads appear to track sentence structure. Treat the weights as one mixing step in one head of one layer, not an account of why the model answered as it did.

In short

Attention keeps every input and lets each step ask its own question of them. The question is a query, each input answers with a key, and a dot product scores the match. A softmax turns the scores into shares summing to one, and the output is the values mixed in those shares. The √d keeps the scores small enough that the softmax stays soft.