How much GPU memory does fine-tuning need?

Out-of-memory errors are the most common reason a first fine-tuning run fails, and the fix is usually arithmetic, not trial and error. Training memory has four parts — weights, gradients, optimizer state and activations — and each scales differently with the method you choose. Here is the accounting, the formula, and a computed table for common model sizes.

The per-parameter accounting#

Mixed-precision training with Adam keeps, for every trainable parameter, a bf16 weight (2 bytes), a bf16 gradient (2 bytes), and in fp32 a master copy of the weight plus Adam's first and second moments (4 + 4 + 4 = 12 bytes). That is the "16 bytes per parameter" figure from the ZeRO paper (Rajbhandari et al., 2019):

For a 70B model that is about 1.1 TB before a single activation is stored. LoRA freezes the base weights and trains low-rank adapters with $P_a\ll P$ parameters, so only the adapters carry gradients and optimizer state (Hu et al., 2021):

QLoRA stores the frozen base in 4-bit NF4 with double-quantised scales (Dettmers et al., 2023); we use 0.55 bytes per parameter to include quantisation constants and a few layers kept in higher precision. We take $P_a=0.5\%P$, which corresponds to rank-16 to rank-64 adapters on all linear layers for these sizes.

Activations#

Korthikanti et al. (2022) give the activation memory of one transformer layer, without recomputation, as

for sequence length $s$, micro-batch $b$, hidden size $h$ and $a$ attention heads. The second term is the attention score matrix; FlashAttention-style kernels never materialise it, leaving about $34sbh$. With activation checkpointing you store only each layer's input ($2sbh$ bytes) and recompute the rest during the backward pass, so total activation memory is roughly $2sbhL$ plus one live layer. We assume checkpointing, FlashAttention, $s=4096$ and micro-batch 1.

Computed table#

Unsharded totals in GB. "Min. H100" assumes ZeRO-3 / FSDP sharding of weights, gradients and optimizer state across GPUs (activations not sharded) and leaves ~10% headroom per GPU; counts are rounded up to powers of two. Architecture shapes are approximate public configurations for dense models of each size.

How to read the numbers#

• Full fine-tuning is an optimizer-state problem. At 70B, 847.2 of 1136.1 GB is Adam state. Sharding it across GPUs is what ZeRO stage 1–3 and FSDP do; 8-bit optimizers cut the 12 bytes to about 6.
• LoRA is a weights problem. At 70B, 141.9 GB of the 153.4 GB total is the frozen model. Sharding the frozen weights (FSDP with LoRA) or quantising them (QLoRA) are the two levers.
• Activations are small with checkpointing at 4k context — under 7 GB here — but scale linearly with sequence length and batch. At 32k context and micro-batch 4, multiply by 32.
• QLoRA puts 70B on one GPU (51 GB), consistent with Dettmers et al.'s 65B-on-48GB result. The price is slower steps and a small quality cost from training against a quantised base.

Long context changes the picture#

Activation memory grows linearly with sequence length even with checkpointing, while weights and optimizer state don't change. For the 70B architecture above with micro-batch 1:

At 32k tokens QLoRA no longer fits on an 80 GB GPU, and the activation term becomes the reason to shard or to use sequence/context parallelism, which splits activations across GPUs along the sequence dimension (Korthikanti et al., 2022). If your training data is long documents, size hardware from this table, not from the 4k one.

RL adds an inference footprint#

GRPO and PPO generate rollouts during training, so the same job also needs inference memory: the KV cache. For grouped-query attention with $n_{kv}$ key/value heads of dimension $d_h$, the cache for one sequence is

For a 70B-class model with 80 layers, 8 KV heads of dimension 128 and 4k-token rollouts, that is about 1.34 GB per sequence; a group of 16 rollouts for 32 prompts is 687 GB of cache on top of everything else. This is why RL jobs usually run generation on separate inference workers (vLLM or SGLang) and why memory bandwidth, not FLOPs, often bounds their speed — see H100 vs H200 vs B200.

Fitting a run you thought wouldn't fit#

1. Turn on activation checkpointing and FlashAttention before anything else.
2. Reduce micro-batch to 1 and use gradient accumulation to keep the effective batch.
3. Switch Adam to an 8-bit or paged variant if you're doing full fine-tuning.
4. Shard with FSDP / ZeRO-3 across all GPUs in the node.
5. Move to LoRA, then QLoRA, if the above still doesn't fit — and evaluate whether quality holds on your task.

Hardware choice follows from the table: see H100 vs H200 vs B200 for fine-tuning. For adapters vs full weights on quality, see LoRA vs full fine-tuning and the LoRA glossary entry.

Frequently asked questions#

References#

1. Rajbhandari, S., Rasley, J., Ruwase, O., He, Y. (2019). ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. arXiv:1910.02054. arxiv.org/abs/1910.02054
2. Korthikanti, V. et al. (2022). Reducing Activation Recomputation in Large Transformer Models. arXiv:2205.05198 (MLSys 2023). arxiv.org/abs/2205.05198
3. Hu, E. J. et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685. arxiv.org/abs/2106.09685
4. Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314. arxiv.org/abs/2305.14314

Related reading