Multi-LoRA serving: S-LoRA, Punica and adapter batching

Multi-LoRA serving runs many LoRA fine-tunes of the same base model on one set of GPUs, in the same batch, at almost the cost of serving the base model alone. The base weights are loaded once. Each request names an adapter; the engine computes the shared base matmul for every token in the batch together, then adds each token's own low-rank correction $B_a{A}_{a}x$ with a kernel that handles many different adapters in one launch. Adapters that are not in use live in CPU memory or on disk and are paged in on demand. This is the serving-side payoff of parameter-efficient post-training: if you fine-tune one adapter per customer, task or experiment, multi-LoRA serving is what makes a thousand fine-tunes cost roughly one deployment rather than a thousand.

The bottleneck: merged models do not batch together#

The obvious way to serve a LoRA fine-tune is to merge it, $W^{\prime }=W+\gamma BA$, and deploy a normal model (see merge or serve the adapter). With $N$ fine-tunes that means $N$ full copies of the weights. Memory is the first problem: an 8B model is 16 GB in bf16, so ten merged fine-tunes need 160 GB before any KV cache. The second problem is subtler and is the reason multi-LoRA kernels exist. Decode is memory-bound (see the roofline): one step costs roughly the time to read the weights once, whatever the batch size up to the ridge point. With separate merged models, a batch of 64 requests spread over $n$ fine-tunes must read $n$ different weight sets, so step time grows linearly in $n$ and the batching that makes LLM serving efficient (see continuous batching) is lost. Unmerged adapters fix both: for token $i$ routed to adapter $a(i)$,

The first term is one dense GEMM over the whole batch, reading $W$ once. The second term reads only the adapters present in the batch. Per layer, the extra bytes from $n$ distinct adapters of rank $r$, relative to the base weight, are

For a 4096×4096 projection at $r=16$ that is 0.78% per adapter. The extra FLOPs, $2r(d_{\text{in}}+d_{\text{out}})$ per token per layer, are similarly small next to the base $2d_{\text{in}}{d}_{\text{out}}$. The cost of multi-tenancy is moved from "one model per tenant" to "a fraction of a percent per tenant per step", but only if the kernel can compute the second term for many adapters at once.

Where it came from#

Two systems papers from late 2023 defined the field. Chen, Ye, Wu, Zhuo, Ceze and Krishnamurthy's Punica introduced the Segmented Gather Matrix-Vector multiplication (SGMV) CUDA kernel, which batches LoRA computation for different adapters in one launch, and a scheduler that consolidates multi-tenant LoRA workloads across a GPU cluster; they reported 12x higher throughput than contemporary serving systems while adding about 2 ms of latency per token. Sheng, Cao, Li, Hooper, Lee and colleagues' S-LoRA stored all adapters in host memory, fetched the ones needed by running requests, and introduced Unified Paging, a single paged memory pool shared by adapter weights of different ranks and KV-cache blocks of different lengths, extending the PagedAttention idea to adapters. It served thousands of adapters on one GPU, with up to 4x the throughput of the then-naive LoRA support in Hugging Face PEFT and vLLM. Follow-up work tackled the remaining problems: dLoRA (Wu et al., OSDI 2024) merges and unmerges adapters dynamically when traffic is skewed toward one adapter and migrates requests and adapters across replicas, and CaraServe (Li et al., 2024) starts prefill of a cold adapter on the CPU while its weights are copied to the GPU, with rank-aware scheduling for latency SLOs. The open-source engines vLLM, SGLang and TensorRT-LLM all ship multi-LoRA serving built on these ideas.

How the kernels work#

The LoRA term is computed in two thin steps, a shrink $v_i=A_{a(i)}{x}_i\in {\mathbb{R}}^r$ and an expand $y_i+=\gamma B_{a(i)}{v}_i$. Two kernel shapes dominate. BGMV (batched gather matrix-vector) treats each token independently, gathering the adapter weights indexed by $a(i)$; it is natural for decode, where each request contributes one token. SGMV (segmented gather) sorts or groups the tokens so that each contiguous segment shares an adapter, then runs one small GEMM per segment inside a single launch, which reuses each adapter's weights across the segment and suits prefill, where one request contributes hundreds of tokens. S-LoRA's kernels add support for heterogeneous ranks in the same batch without padding everything to the largest rank. Engines usually precompute, per step, a small index tensor mapping tokens to adapter slots, and pass it to both kernels.

Memory management is the other half. Adapter weights for the active set sit in a GPU pool with a fixed number of slots (vLLM's max_loras) or, in S-LoRA, in pages of the same unified pool that holds KV-cache blocks, so a burst of adapters and a burst of long contexts compete for the same memory without fragmentation. Inactive adapters live in host RAM (an LRU cache of configurable size) and, beyond that, on disk or object storage. When a request arrives for an adapter that is not resident, the engine copies it over PCIe and the request waits, or, in CaraServe's design, starts on the CPU.

Worked example: a thousand adapters on one 8B base#

Take a Llama-3-8B-shaped model: 32 layers, hidden size 4096, grouped-query K/V projections of width 1024, MLP width 14,336, LoRA on all seven linear projections. One layer's $\sum (d_{\text{in}}+d_{\text{out}})$ is $8192+5120+5120+8192+3\times 18432=81,920$, so one adapter has $32\times 81920\times r=2.62\text{M}\times r$ parameters. At $r=16$ that is 41.9M parameters, 83.9 MB in bf16, against 6.98B parameters (about 14 GB) in the base linear layers.

Memory. One thousand rank-16 adapters are 84 GB, so they cannot all sit on one 80 GB GPU next to the base and a KV cache, but 64 resident slots cost 5.4 GB and host RAM holds the rest easily. Load time. Copying one adapter at an effective 25 GB/s over PCIe takes 3.4 ms; at rank 64, 13.4 ms. Both are small next to a typical prefill, which is why host-memory paging works; loading from network storage is a different story and needs prefetching. Decode overhead. On an H100-class GPU with 3.35 TB/s, reading the base linear weights takes about 4.2 ms per step; each distinct rank-16 adapter adds $83.9\text{MB}/3.35\text{TB/s}\approx 25\mu \text{s}$. A batch with 44 distinct adapters adds about 1.1 ms, or 26%. That is the honest cost of heterogeneity: modest, but proportional to the number of distinct adapters and to rank.

Measured and simulated numbers#

A kernel-shape benchmark on a laptop

We timed one 4096×4096 fp16 projection with rank-16 adapters on an Apple-silicon Mac (PyTorch 2.10, MPS backend), 64 decode tokens spread round-robin over $n$ distinct adapters, median of 60 calls after warm-up, checking that all methods produce the same output. Four strategies: one merged weight copy per adapter (the "deploy separate fine-tunes" baseline, restricted to the tokens of each adapter), the shared base GEMM plus a Python loop of per-adapter LoRA matmuls, the shared base GEMM plus one gathered batched matmul (a BGMV-style computation using torch.bmm on gathered $A$ and $B$), and the base GEMM alone.

The single-launch batched form stayed between 1.3 and 1.7 ms from 1 to 64 adapters, while both looped forms grew roughly linearly to about 9 ms at 64, five times slower. Two caveats keep this honest. On MPS a large part of the looped cost is per-launch overhead rather than memory traffic, so the slope is steeper than it would be with CUDA graphs on a GPU. And our gathered BMM materialises per-token copies of the adapter weights, so its fixed overhead (about 0.7 to 1 ms over the base GEMM here) is much larger than a fused BGMV or SGMV kernel's; Punica reports about 2 ms per token end to end for the whole model. The shape of the result is the point: whether the per-adapter work happens in one launch or $n$ launches decides whether heterogeneity is nearly free or linear.

How many adapters are actually in a batch?

Adapter traffic is usually skewed: a few adapters get most requests. We simulated 1,000 adapters with Zipf popularity $p_k\propto k^{-s}$ (numpy, seed 0), drew batches of 64 concurrent requests, and replayed 200,000 arrivals through an LRU cache of GPU slots, counting hits after a 20,000-request warm-up.

Even under strong skew, a batch of 64 contained 33 to 62 distinct adapters, so engines must be built for highly heterogeneous batches, not occasional mixing. And GPU slot hit rates are low with a long tail: at 64 slots and Zipf 1.0, half of arrivals need a host-to-GPU copy. That is acceptable at 3.4 ms per rank-16 copy, but it means the host RAM tier, not the GPU tier, is what must hold the working set, and anything slower than host RAM (a disk or network fetch of tens to hundreds of milliseconds) lands directly in time to first token.

What the engine does each step#

on request arrival(req):
    a = req.adapter
    if a not in gpu_pool:
        if a not in host_cache: host_cache.load_from_storage(a)      # slow path: disk or network
        slot = gpu_pool.evict_lru_unused()                           # never evict an adapter in use
        async_copy(host_cache[a] -> gpu_pool[slot])                  # ~ms over PCIe
    scheduler.enqueue(req, ready_when=copy_done(a))

each step:
    batch = scheduler.pick(max_tokens, max_distinct_adapters=max_loras)
    idx = [gpu_pool.slot_of(r.adapter) for each token in batch]      # token -> adapter slot
    for layer in model:
        for proj in layer.lora_targets:
            y = x @ W_proj.T                                         # one dense GEMM
            v = shrink(x, A_pool[proj], idx)                         # BGMV (decode) / SGMV (prefill)
            y += expand(v, B_pool[proj], idx, scaling)
    sample; release finished requests' KV blocks and adapter references

The scheduler constraint max_distinct_adapters is the multi-LoRA analogue of a batch-size cap: a request whose adapter cannot get a slot waits even if there is KV-cache room for it. Fair schedulers also stop one hot adapter from starving the tail, and some (dLoRA) switch to merging an adapter into a dedicated replica when it dominates traffic.

Interaction with how the adapters were trained#

Same base, exactly. Every adapter must have been trained on the same base checkpoint, at the same revision. An adapter trained on an instruct model and served on its base model is wrong, even though the shapes match. Quantized bases. Multi-LoRA engines commonly run a quantized base (INT4, FP8) with 16-bit adapters. An adapter trained on the bf16 base then runs on a perturbed base; our measurements in the GPTQ and AWQ article show this kept only 64 to 73% of a style fine-tune's log-likelihood gain at INT4, so either train adapters on the quantized base (QLoRA on the same quantized checkpoint) or evaluate each adapter on the serving base. Ranks and targets. Engines pad to a maximum rank per pool (max_lora_rank), so one rank-256 adapter can inflate every slot; keep ranks uniform within a deployment where possible. Adapters that also train embeddings, the LM head or add new tokens are restricted or unsupported in many engines; keep tokenizer changes out of adapters you intend to multiplex. Variants. DoRA adds a per-column magnitude rescaling that standard BGMV and SGMV kernels do not compute; IA3 and prompt tuning need different kernels again. Plain LoRA, including LoRA+ and PiSSA-initialised adapters after conversion, is the portable format. Chat templates and system prompts belong to the request, not the adapter, so a client must send each adapter the template it was trained with. Prefix caching keys must include the adapter ID, because the same prompt produces different KV entries under different adapters (see prefix caching).

Tuning knobs#

When to use it, and what to use instead#

Failure modes and how to detect them#

Cold-start spikes in TTFT. Requests for non-resident adapters wait for a copy, and if the host cache is too small, for storage. Detect by splitting TTFT percentiles by "adapter was resident" versus "not"; fix with a larger host cache or prefetching. Slot starvation. With max_loras too small, requests queue for slots while GPU memory sits idle; queue time grows with the number of distinct adapters. Rank padding waste. One high-rank adapter forces every slot to its size; memory per slot is visible in engine logs. Silent wrong base. An adapter loaded against a different base revision produces fluent but degraded output; record the base model hash in adapter metadata and check it at load. Quality drift on a quantized base, invisible unless each adapter has its own regression eval on the serving configuration. Throughput collapse from heterogeneity when an engine falls back to per-adapter loops (for example, an unsupported adapter type); watch tokens per second as a function of distinct adapters per batch, which should be nearly flat.

Implementation sketch#

With vLLM, adapters are named at launch or loaded at runtime, and requests select one by name; SGLang's --lora-paths and TensorRT-LLM's LoRA support follow the same model. In POFE, every LoRA recipe exports an adapter with its base checkpoint recorded, so it can be dropped into a shared pool like this.

# server
# vllm serve BASE_MODEL --enable-lora --max-loras 32 --max-lora-rank 16 \
#     --max-cpu-loras 512 --lora-modules support=/adapters/support legal=/adapters/legal

# client: the OpenAI-compatible API picks the adapter by model name
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
r = client.chat.completions.create(model="legal",
        messages=[{"role": "user", "content": "Summarise clause 4."}])

# offline, many adapters in one batch
from vllm import LLM, SamplingParams
from vllm.lora.request import LoRARequest
llm = LLM(model=BASE_MODEL, enable_lora=True, max_loras=8, max_lora_rank=16)
reqs = [LoRARequest(name, i + 1, path) for i, (name, path) in enumerate(adapters)]
outs = llm.generate(prompts, SamplingParams(max_tokens=128),
                    lora_request=[reqs[k] for k in adapter_of_prompt])

The math of a BGMV-style layer, as in our benchmark, fits in three lines of PyTorch; production kernels compute the same thing without materialising the gathered weights:

def multi_lora_linear(x, W, A_pool, B_pool, idx, scale):   # x: [T, d_in], idx: [T]
    y = x @ W.T                                             # shared base GEMM
    v = torch.bmm(A_pool[idx], x[:, :, None])               # shrink: [T, r, 1]
    return y + scale * torch.bmm(B_pool[idx], v).squeeze(2) # expand: [T, d_out]

Further reading#

Chen et al. (2023) on Punica for the SGMV kernel and cluster scheduling; Sheng et al. (2023) on S-LoRA for unified paging, heterogeneous-rank kernels and tensor-parallel LoRA; Wu et al. (2024) on dLoRA for dynamic merging and migration; Li et al. (2024) on CaraServe for CPU-assisted cold starts and SLO-aware scheduling. Hu et al. (2021) is the LoRA paper itself. For the single-adapter decision, read merge the LoRA or serve the adapter; for the economics of sharing a deployment across tenants, see utilization and batching economics.

Frequently asked questions#

References#

1. Lequn Chen, Zihao Ye, Yongji Wu, Danyang Zhuo, Luis Ceze, Arvind Krishnamurthy (2023). Punica: Multi-Tenant LoRA Serving. arxiv.org/abs/2310.18547
2. Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, et al. (2023). S-LoRA: Serving Thousands of Concurrent LoRA Adapters. arxiv.org/abs/2311.03285
3. Bingyang Wu, Ruidong Zhu, Zili Zhang, Peng Sun, Xuanzhe Liu, Xin Jin (2024). dLoRA: Dynamically Orchestrating Requests and Adapters for LoRA LLM Serving. www.usenix.org/conference/osdi24/presentation/wu-bingyang
4. Suyi Li, Hanfeng Lu, Tianyuan Wu, Minchen Yu, Qizhen Weng, et al. (2024). CaraServe: CPU-Assisted and Rank-Aware LoRA Serving for Generative LLM Inference. arxiv.org/abs/2401.11240
5. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arxiv.org/abs/2106.09685
6. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, et al. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. arxiv.org/abs/2309.06180

Related reading