Low-rank adaptation: customising a huge model by training almost none of it
To teach a huge language model a new task, you can leave every one of its weights alone and train a small correction beside each weight matrix instead: two thin matrices whose product is the change. On GPT-3, with 175 billion weights, this trained up to ten thousand times fewer numbers than ordinary fine-tuning and matched or beat it on the tasks the authors tried.
The cost of a full copy
A model trained on general text is usually adapted to a specific job by fine-tuning: keep training it on examples of that job, and let every weight move. The result is a whole new model, as large as the original. For GPT-3 that is 175 billion numbers per task. A company that wants a summariser, a question answerer and a translator from text into database queries ends up storing and serving three giant models that differ only slightly from one another. Training is expensive too, because an optimiser like Adam keeps extra running values for every weight it updates, so memory grows with the number of weights being trained.
The idea: train only the change, and keep it thin
Take one weight matrix in the network, with d rows and k columns. Fine-tuning would end with that matrix plus some change. Low-rank adaptation, or LoRA, proposed by Edward Hu and colleagues in 2021, freezes the original matrix and writes the change as the product of two smaller matrices: B, with d rows and r columns, and A, with r rows and k columns. The number r, the rank, is tiny compared with d and k. The input goes through the frozen matrix and through the pair, and the two outputs are added. Only A and B are trained.
The saving is in the arithmetic. A full change to the matrix has d × k numbers. The pair has r × (d + k). In GPT-3 the attention matrices are 12,288 by 12,288, so a full change is about 151 million numbers, and a rank-4 pair is under 100,000.
Interactive The top-left grid is a made-up change to one 24 × 24 weight matrix: three strong patterns plus some noise. Drag rank r to choose how thin the pair B and A may be, and watch how much of the change their product BA can carry, and how many numbers it takes. Drag noise to make the change less tidy, and press new change for different patterns.
Why a thin change should be enough
The product of a d-by-r and an r-by-k matrix can only push its inputs along r independent directions, however many entries it has. So LoRA is a bet: that adapting a model to one task only needs a few directions per matrix. The authors took the hint from earlier work showing that large pre-trained models can still learn a new task when training is confined to a small random slice of their possible changes. They then checked the bet. On GPT-3 a rank of one or two already did well. When they trained the same matrix at rank 8 and at rank 64 and compared what was learned, the leading directions of the two runs largely coincided, and the authors judged that the rest of the rank-64 version was probably mostly noise picked up in training.
They also asked what the change does. It does not repeat the directions the original matrix already uses most. It picks out directions the original matrix holds only weakly and enlarges them, by a factor of about 20 at rank 4. Their reading is that pre-training already learned the features a task needs, and adaptation mostly turns up the ones that task cares about.
Starting from the original model, ending with no extra cost
Two details make the method practical. First, A starts random and B starts at zero, so their product starts at zero: on step one the model behaves exactly like the pre-trained one, and training moves it away from there. Second, once training is done, the product BA can be added into the frozen matrix. The model then runs at exactly its original size and speed. An older approach, the adapter, inserts small extra layers that every input must pass through in sequence, and the authors measured a noticeable slowdown from them when requests arrive one at a time. To switch tasks with LoRA, subtract one product and add another.
What it bought
On GPT-3, adapting only the query and value matrices of attention at rank 4 shrank the saved result for a task from about 350 GB to about 35 MB. A hundred tasks then fit in about 354 GB rather than 35 TB. Memory during training fell from 1.2 TB to 350 GB, since the optimiser no longer tracks the frozen weights, and training ran about 25% faster. On RoBERTa, DeBERTa, GPT-2 and GPT-3, the quality matched or beat full fine-tuning. With a fixed budget of trainable numbers on GPT-3, putting all of it into the query matrices, or all into the key matrices, at rank 8 did markedly worse than splitting it between query and value matrices at rank 4.
Where it may not be enough
The authors do not claim a small rank fits every task. Their own example: if the new task were in a language the model had never read, retraining everything could beat a thin correction. And if the corrections are merged into the weights for speed, one batch of inputs cannot easily mix requests for different tasks. Why a few directions are enough is, by their account, still far from clear.
In short
Freeze the giant model. Beside each chosen weight matrix, train two thin matrices whose product is the change, starting from zero. The change can only move along a few directions, and for the tasks tested a few were enough. Each task becomes a file of megabytes instead of hundreds of gigabytes, and once merged it costs nothing extra to run.
Where this comes from
- LoRA: Low-Rank Adaptation of Large Language Models linked only, not reproduced
arxiv.org/abs/2106.09685