Merge the LoRA or serve the adapter?

A LoRA fine-tune can reach production in two forms. Merged, the low-rank update is added into the base weights once, $W^{\prime }=W+\gamma BA$, and the result is an ordinary checkpoint with exactly the base model's shape, speed and serving stack. Unmerged, the base stays untouched and the adapter's two thin matrices are applied at runtime on every forward pass, either for one adapter or for many at once with multi-LoRA serving. Mathematically the two are the same function. In practice they differ in latency, memory, numerical precision, how quantization interacts with the fine-tune, how fast you can switch or roll back, and how many GPUs a portfolio of fine-tunes costs. This article puts numbers on each, and ends with a decision table. It is the last step of a LoRA-based post-training pipeline, after SFT, DPO or RL have produced an adapter and before anything is deployed.

What each form costs at inference time#

For a projection with weight $W\in {\mathbb{R}}^{d_{\text{out}}\times d_{\text{in}}}$, adapter $A\in {\mathbb{R}}^{r\times d_{\text{in}}}$, $B\in {\mathbb{R}}^{d_{\text{out}}\times r}$ and scaling $\gamma$ ($\alpha /r$ in standard LoRA, $\alpha /\sqrt{r}$ in rsLoRA), the two forms compute

The merged form has zero runtime overhead by construction. The unmerged form adds three costs per adapted projection per step. FLOPs: $2r(d_{\text{in}}+d_{\text{out}})$ per token, a fraction $r(d_{\text{in}}+d_{\text{out}})/(d_{\text{in}}{d}_{\text{out}})$ of the base, 0.78% for a 4096×4096 projection at $r=16$. Bytes: the same fraction of the base weight bytes, read once per step per distinct adapter. Since decode is memory-bound (see the roofline), this is the term that shows up in throughput. Kernel launches: two extra matmuls and an add per projection unless the engine fuses them. A 32-layer model with adapters on all seven projections gets $32\times 7\times 2=448$ extra matmuls per step. At batch 1 an 8B model's step is about 5 ms of weight reading on an H100-class GPU, and a few microseconds of launch and synchronisation overhead per tiny kernel can add up to a visible fraction of that unless the engine captures the step in a CUDA graph or fuses the LoRA into the base GEMM's epilogue. The overhead ratio of unmerged serving is therefore roughly

with $k$ extra kernels per step and ${\tau }_{\text{launch}}$ their fixed cost. The first term is small for one adapter and grows with heterogeneity; the second is an implementation detail that good engines drive toward zero and naive ones do not. Punica reported about 2 ms of added latency per token for its batched multi-adapter kernels.

Measured on a laptop

We served Qwen2.5-0.5B-Instruct in fp16 on an Apple-silicon Mac (PyTorch 2.10, MPS) three ways: the base model, the base with our rank-16 style adapter merged, and the base with the same adapter applied unmerged as two extra matmuls on all 168 projections (24 layers × 7), using Hugging Face generate for 64 new tokens. The machine was shared with other heavy jobs (load average around 90 to 115), so we interleaved the three models over four rounds and report the best round.

Base and merged run identical computations, so their 3 to 17% differences are noise. Against that noise the unmerged model was 42 to 48% slower in three of four batch sizes, and an earlier, uninterleaved run of the same script showed the same ordering (65.9 ms merged against 97.6 ms unmerged at batch 1). This is the launch-overhead term at work: in eager PyTorch the 336 extra small matmuls per step are not free, even though their FLOPs and bytes are under 2% of the base. A GPU engine with fused LoRA kernels and graph capture removes most of this, which is why the measured gap in production engines is much smaller; the measurement shows what you get without that engineering.

Where the two forms come from#

Hu et al. (2021) designed LoRA to be mergeable: the update is linear in the frozen weight, so after training it can be folded in and "introduces no inference latency", the paper's argument against adapter layers inserted in series. The same paper noted the flip side: merging means one model per task, so batching inputs from different tasks in one forward pass is no longer straightforward. Systems work took up the unmerged form for that reason. Punica (Chen et al., 2023) and S-LoRA (Sheng et al., 2023) built kernels and memory managers to serve many unmerged adapters in one batch, and dLoRA (Wu et al., OSDI 2024) showed that neither form wins everywhere: when traffic is skewed toward one adapter it pays to merge it, and when traffic is mixed it pays to unmerge, so dLoRA switches dynamically. QLoRA (Dettmers et al., 2023) added a precision dimension to the choice, because an adapter trained on a 4-bit base is not a correction to the original bf16 weights.

The numerics of merging#

A merge is a floating-point addition, and the update is small relative to the weights. A bf16 number has 8 significant bits, so the spacing between representable values near $w$ is

and any update entry smaller than half a ulp disappears when the merged weight is rounded back to bf16. Worked by hand: a weight stored as $0.0517578$ in bf16 has ulp $2^{-5-7}=0.000244$. Adding $+0.0003$ rounds to $+0.000244$ (19% error); adding $+0.0001$ rounds to zero; adding $-0.00015$ rounds to $-0.000244$ (63% error). A weight of $-0.20996$ has ulp 0.000977, so an update of $+0.0004$ vanishes and $+0.0009$ becomes $+0.000977$. We checked these with PyTorch's bfloat16.

On the real adapter, merging into bf16 weights zeroed 19.0% of the update's entries and the effective update differed from the intended one by 14.6% in Frobenius norm; merging into fp16 zeroed 2.6% with a 1.9% error. Behaviourally this did not matter: the bf16-merged model scored $-0.634$ nats per token on the fine-tune's held-out targets against $-0.636$ for the fp32 merge, and produced the trained style on the same 93.3% of 60 held-out prompts (measured in our quantization experiment). Rounding errors are close to unbiased and uncorrelated, so they average out over thousands of weights feeding each output. Two cases where they do not: very small learning rates or very low ranks, where most of the update sits below half a ulp and is systematically erased, and repeated merges (merging adapter after adapter into bf16), where errors accumulate. The cheap insurance is to merge in fp32 and cast once.

Merging and quantization

If the deployment is quantized, the order of operations matters more than the precision of the addition. Three cases. Adapter trained on a bf16 base, served quantized. Merge in fp32, then quantize the merged model with calibration data from the fine-tune; in our measurements this kept 90% of the fine-tune's log-likelihood gain with GPTQ at INT4, against 64 to 73% for quantizing the base and serving the exact adapter unmerged on top. The adapter was a correction to the bf16 base and is not a correction to its quantized version. Adapter trained with QLoRA on a 4-bit base. The adapter corrects the dequantized base $\tilde{W}$, not $W$. Either serve it unmerged on the same 4-bit base, or merge into $\tilde{W}$ dequantized to bf16 and serve in bf16; merging into the original $W$ produces a model nobody trained. A quantized deployment with no adapter overhead. QA-LoRA (Xu et al., 2023) constrains the adapter so it folds into the INT4 grid exactly, avoiding the re-quantization step.

The operational trade-offs#

Artefact size. A rank-16 all-linear adapter for an 8B model is 41.9M parameters, 84 MB in bf16; the merged model is 16 GB. One hundred fine-tunes are 8.4 GB of adapters or 1.6 TB of merged checkpoints. Cold start. At 2 GB/s from local NVMe, an adapter loads in about 40 ms and a merged 8B model in about 8 s, before any engine warm-up; scale-to-zero is practical for adapters and painful for merged models. Switching and rollback. Merging in place is cheap (our whole 0.5B model took 18 to 26 ms on MPS, it scales with parameter count), and unmerging is the same subtraction, but in-place merge and unmerge in bf16 is not exactly reversible because of the rounding above, so keep a pristine base copy rather than subtracting. With unmerged serving, rollback is changing which adapter name a route points to. A/B tests and canaries are trivial with adapters: two adapters on one pool, split traffic by request. With merged models they are two deployments. Isolation and audit. A merged checkpoint is a single, hashable artefact with a clear provenance, which some compliance regimes prefer; a pool of adapters on a shared base needs base-hash checks at load time. Engine support. Every serving engine and every kernel (quantized, speculative, tensor-parallel) supports a plain checkpoint. LoRA support varies: speculative decoding drafts, some quantization formats and some adapter variants may not combine with adapters in a given engine version. Variants. DoRA merges cleanly (its magnitude vector folds into the merged weight) but is often unsupported unmerged; IA3 merges into a column scaling; adapters that also train embeddings or modules_to_save need those tensors replaced on merge.

The portfolio view

The cost comparison changes sign with the number of fine-tunes. Suppose (as an assumption, not a benchmark) one GPU sustains 5,000 decode tokens per second for one model at your latency SLO, and fine-tuned traffic totals 20,000 tokens per second across $N$ adapters with Zipf(1.0) popularity. Merged, every adapter needs at least one GPU plus replicas for its share; pooled, the total traffic needs $20000/(5000/(1+0.006n_{\text{distinct}}))$ GPUs, using 0.6% extra bytes per distinct rank-16 adapter in a batch for 8B shapes.

With one adapter both need 4 to 5 GPUs, and merged wins slightly because it has no overhead. From 8 adapters on, merged needs 9, then 17, 32, 64, 128 and 256 GPUs, because every long-tail adapter occupies a GPU it cannot fill; the pool stays at 5 to 6. A hybrid that gives merged replicas only to adapters whose traffic fills a GPU by itself, and pools the rest, matched the pool here. The exact numbers depend on the assumed throughput; the crossover at a handful of adapters does not.

Inside training loops

The same choice appears before deployment, in online post-training. In GRPO, online DPO or on-policy distillation with LoRA, a fast inference engine generates rollouts from the current policy, and the policy changes every step. Pushing an unmerged adapter to the sampler moves 84 MB per sync for an 8B model; pushing merged weights moves 16 GB and forces the engine to reload them. That is why LoRA-based RL setups usually serve the adapter unmerged to the sampler, accept a small per-token overhead, and merge only once training ends. The precision caveat carries over: if the sampler runs a quantized base under the adapter, the rollouts come from a slightly different policy than the one being updated, and the importance ratios should account for it.

How to decide#

The merge, step by step#

for each adapted module (q, k, v, o, gate, up, down in every layer):
    W32   = base_weight.float()                      # never add in bf16
    W_src = dequantize(base_4bit) if adapter_was_qlora else W32
    delta = scaling * (B.float() @ A.float())        # scaling = alpha / r  (alpha / sqrt(r) for rsLoRA)
    W_new = W_src + delta                            # DoRA: renormalise columns and apply magnitude here
    store W_new.to(serving_dtype)
replace embeddings / lm_head if the adapter trained them (modules_to_save)
save a full checkpoint + record base hash and adapter hash
evaluate the merged artefact against the unmerged adapter on the fine-tune's held-out set
optionally quantize the merged model, calibrated on fine-tune data, and evaluate again

The comparison step catches nearly every merge bug: the wrong scaling (forgetting rsLoRA's $\sqrt{r}$), a missing module, a double merge, or a QLoRA adapter merged into the wrong base. The merged and unmerged models should agree to within numerical noise on held-out prompts, for example a per-token KL below about $10^{-3}$ and identical greedy outputs on most prompts.

Failure modes and how to detect them#

Silent scaling errors. A merged model that behaves like a weaker version of the fine-tune usually has $\gamma$ wrong; compare against unmerged on held-out data. QLoRA merge mismatch. The merged model is noticeably worse than the unmerged adapter on its 4-bit base; merge into the dequantized base instead. Precision erosion. Merging tiny updates into bf16 erases the smallest entries; harmless in our test, but check when learning rates are low or when merging repeatedly, and merge in fp32 anyway. Unmerged latency regressions. Tokens per second drops more than the bytes formula predicts; the engine is not fusing LoRA kernels or is not graph-capturing the step, as in our eager-mode measurement. Adapter on the wrong base. Same shapes, different revision; output is fluent and wrong. Store and check the base hash. Quantization after merge erases the fine-tune. Covered in detail in GPTQ and AWQ: calibrate on fine-tune data and measure the fine-tune's own metric.

Implementation sketch#

With the open-source PEFT library, merge_and_unload performs the merge; loading the base in fp32 first makes the addition exact to fp32 precision. In POFE, LoRA recipes can export either form, and the evaluation runs on whichever form will be served.

import torch
from transformers import AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(BASE_DIR, torch_dtype=torch.float32)   # fp32 for the add
model = PeftModel.from_pretrained(base, ADAPTER_DIR)
merged = model.merge_and_unload()                                    # W + scaling * B @ A
merged.to(torch.bfloat16).save_pretrained(MERGED_DIR, safe_serialization=True)

# sanity check: merged vs unmerged on held-out prompts (both in fp32)
@torch.no_grad()
def kl_per_token(p_model, q_model, ids):
    lp = torch.log_softmax(p_model(ids).logits.float(), -1)
    lq = torch.log_softmax(q_model(ids).logits.float(), -1)
    return (lp.exp() * (lp - lq)).sum(-1).mean().item()

# serve merged: vllm serve MERGED_DIR
# serve unmerged: vllm serve BASE_DIR --enable-lora --lora-modules ft=ADAPTER_DIR

Further reading#

Hu et al. (2021) for LoRA and the original no-latency merge argument; Kalajdzievski (2023) for rsLoRA's $\alpha /\sqrt{r}$ scaling, which must be used at merge time too; Liu et al. (2024) for DoRA, whose magnitude vector changes the merge formula. Dettmers et al. (2023) on QLoRA and Xu et al. (2023) on QA-LoRA for adapters and quantized bases. Wu et al. (2024) on dLoRA for dynamic merge and unmerge under skewed traffic; Chen et al. (2023, Punica) and Sheng et al. (2023, S-LoRA) for batched unmerged serving. Continue with multi-LoRA serving, and for the cost side, self-hosting versus API TCO.

Frequently asked questions#

References#

1. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arxiv.org/abs/2106.09685
2. Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xuanzhe Liu, Xin Jin (2024). dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving. www.usenix.org/conference/osdi24/presentation/wu-bingyang
3. Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, Arvind Krishnamurthy (2023). Punica: Multi-Tenant LoRA Serving. arxiv.org/abs/2310.18547
4. Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, et al. (2023). S-LoRA: Serving Thousands of Concurrent LoRA Adapters. arxiv.org/abs/2311.03285
5. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, Luke Zettlemoyer (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arxiv.org/abs/2305.14314
6. Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, et al. (2023). QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models. arxiv.org/abs/2309.14717
7. Damjan Kalajdzievski (2023). A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA. arxiv.org/abs/2312.03732
8. Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, et al. (2024). DoRA: Weight-Decomposed Low-Rank Adaptation. arxiv.org/abs/2402.09353
9. Hugging Face (2022). PEFT: Parameter-Efficient Fine-Tuning library. github.com/huggingface/peft

Related reading