LoRA (Low-Rank Adaptation)

LoRA (Low-Rank Adaptation): A parameter-efficient fine-tuning method that freezes the base weights and learns a low-rank update ΔW = BA for selected layers.

Explained#

LoRA (Hu et al., 2021) freezes a weight matrix $W\in {\mathbb{R}}^{d\times k}$ and learns an update of rank $r$:

For a 4096×4096 projection at $r=16$, that is 131k trainable parameters instead of 16.8M. Because only the adapters carry gradients and optimizer state, training memory falls from about 16 bytes per parameter to about 2 (see GPU memory for fine-tuning). QLoRA (Dettmers et al., 2023) stores the frozen base in 4-bit, cutting memory roughly 3–4× further.

Trade-offs

Biderman et al. (2024) find LoRA underperforms full fine-tuning on large-scale code and math training but forgets less of the base model's other abilities. For typical SFT, DPO and RL on modest datasets, LoRA on all linear layers with a learning rate around 10× the full fine-tuning rate comes close. Adapters are also cheap to serve: hundreds can share one base model.

References#

1. Hu, E. J. et al. (2021). LoRA. arXiv:2106.09685. arxiv.org/abs/2106.09685
2. Dettmers, T. et al. (2023). QLoRA. arXiv:2305.14314. arxiv.org/abs/2305.14314
3. Biderman, D. et al. (2024). LoRA Learns Less and Forgets Less. arXiv:2405.09673. arxiv.org/abs/2405.09673

Related reading