LoRA vs full fine-tuning: quality, cost and serving trade-offs

LoRA made fine-tuning affordable: instead of updating billions of weights, you train a few million in low-rank adapters and leave the base model frozen. But "LoRA is as good as full fine-tuning" is only sometimes true. This guide covers how LoRA works, the memory maths, which hyperparameters actually matter, where adapters fall short, and the serving advantages that often decide the question.

How LoRA works

Full fine-tuning updates every weight matrix W in the model. LoRA (Low-Rank Adaptation) freezes W and learns an update ΔW = B·A, where A and B are thin matrices with an inner dimension r, the rank. For a 4096×4096 projection, a rank-16 adapter trains about 131,000 parameters instead of 16.8 million — under 1%.

The bet behind LoRA is that the change fine-tuning needs to make is low-rank: the model already has the knowledge, and post-training mostly redirects it. For many tasks — format, style, domain vocabulary, a specific extraction schema — that bet holds well. For tasks that require the model to absorb a lot of new information or substantially change its reasoning, it holds less well.

QLoRA and DoRA

QLoRA stores the frozen base model in 4-bit precision and trains the adapter in higher precision, cutting memory roughly 3–4× further at a small quality and speed cost. It is how a 70B model can be fine-tuned on a single 80 GB GPU. DoRA decomposes weights into magnitude and direction and adapts both, often closing part of the remaining gap to full fine-tuning at similar cost.

The memory maths

Memory, not compute, is usually what decides how you train. With mixed-precision Adam, full fine-tuning needs roughly 16 bytes per parameter before activations: 2 for bf16 weights, 2 for gradients, and 12 for fp32 master weights and two optimizer moments. LoRA needs 2 bytes per frozen parameter plus the small adapter and its optimizer state. QLoRA needs about 0.5–0.6 bytes per frozen parameter.

These figures exclude activations, which scale with batch size and sequence length; gradient checkpointing trades roughly 30% more compute for far less activation memory. Sharding with FSDP or DeepSpeed ZeRO-3 spreads full fine-tuning state across GPUs, so the numbers above become per-cluster rather than per-GPU requirements.

Quality: when LoRA matches full fine-tuning

Published comparisons and our own runs point the same way. LoRA matches or comes within noise of full fine-tuning when:

• The dataset is small to medium (up to a few hundred million tokens).
• The task is mostly about format, style or applying knowledge the model already has.
• The adapter is applied to all linear layers (attention and MLP), not just the query and value projections.
• Learning rate is set for LoRA — typically about 10× higher than for full fine-tuning.
• The method is RL. Because RL updates carry relatively little information per sample, low-rank adapters capture most of the gain; LoRA GRPO often performs almost identically to full-weight GRPO.

Full fine-tuning pulls ahead when:

• You are doing continued pre-training on large new corpora — a new language, a large proprietary codebase, a specialised scientific domain.
• The dataset is large (billions of tokens) and the adapter's capacity becomes the bottleneck.
• You need maximum quality on a single, high-value model and serving cost is secondary.

Hyperparameters that actually matter

For full fine-tuning, start around 1e-5 to 2e-5 with cosine decay and a short warm-up, and expect to need more careful tuning: full-weight runs are less forgiving of a learning rate that's slightly too high.

Common LoRA mistakes

Most disappointing LoRA results trace back to a handful of avoidable errors rather than a fundamental limit of adapters.

• Using the full fine-tuning learning rate. A learning rate of 1e-5 that works well for full-weight training is usually far too low for LoRA, which needs roughly an order of magnitude more. Runs look stable and simply don't learn much.
• Adapting attention only. Early LoRA recipes targeted just the query and value projections. Applying adapters to the MLP layers as well consistently closes much of the gap to full fine-tuning, especially when the task involves new domain knowledge.
• Changing rank without adjusting alpha. The effective scale of the update is alpha divided by rank. Doubling rank while holding alpha fixed halves the update size, so a rank sweep can look flat for the wrong reason.
• Merging into a different precision. Training an adapter on a 4-bit QLoRA base and then merging it into bf16 weights changes the function the adapter was trained against. Evaluate the merged model, not just the adapter in training mode.
• Forgetting the chat template. Adapters learn whatever format the training data uses. If the template at training time differs from the one used in serving — missing system prompt, different special tokens — quality drops sharply in production even though offline evals looked good.
• Evaluating on loss alone. Training and validation loss are poor proxies for task quality, particularly for preference and RL methods. Always evaluate on the task metric you ship on.

None of these require more compute to fix — just a checklist. POFE's recipes set target modules, learning rates, alpha scaling and chat templates per model family by default, which is why first runs on the platform tend to be usable baselines rather than debugging sessions.

Serving: the underrated advantage

The biggest practical argument for LoRA often isn't training cost — it's serving. A LoRA adapter for a 32B model is tens to a few hundred megabytes, versus 64 GB for a full copy. That enables multi-LoRA serving: one copy of the base model in GPU memory, with hundreds of adapters swapped per request. Runtimes like vLLM and SGLang batch requests for different adapters together with modest overhead.

The consequences are significant. A fine-tuned model per customer, per language or per workflow becomes economically viable. Rolling back is swapping an adapter. A/B testing two training runs costs nothing extra in GPUs. With full fine-tuning, every variant is a separate deployment with its own GPUs.

If you do fine-tune full weights, you can still recover some of these benefits by extracting a low-rank approximation of the difference afterwards, at some quality cost.

Our recommendation

1. Default to LoRA on all linear layers, rank 32–64, for SFT, DPO and RL.
2. Use QLoRA when GPU memory is the constraint and a small quality loss is acceptable — prototyping, or very large models on limited hardware.
3. Move to full fine-tuning for continued pre-training, very large datasets, or when a rank sweep shows adapter capacity is limiting.
4. Decide serving architecture early: if you'll have many variants, LoRA's serving economics usually outweigh small quality differences.

For a cost comparison of both approaches see How much does it cost to fine-tune an LLM?, or try both in the interactive demo — switch the adapter on the Method step and watch the GPU requirement change.

Frequently asked questions