Inference 101
One page per topic, grouped by family: the mechanism, the arithmetic, the knobs that matter and how they interact with the models you train.
Memory and attention
The KV cache, attention kernels and where the memory goes.
FlashAttention for inference: tiled prefill and split-KV decoding
FlashAttention tiles attention through on-chip memory with a running softmax so the N x N score matrix is never written to HBM; Flash-Decoding splits the KV cache across thread blocks for fast long-context decode.
Read →Inference 101The KV cache: memory formula, GQA, MQA, MLA and long-context sizing
The KV cache stores each layer's keys and values for every processed token so decoding does one token of work per step, at a memory cost of 2 x layers x KV heads x head dim x bytes per token.
Read →Inference 101KV cache quantization and eviction: FP8, INT4, H2O and attention sinks
KV cache compression stores cached keys and values in fewer bits (FP8, INT4, INT2) or keeps fewer of them (sinks plus recent window, heavy hitters), trading accuracy for memory and bandwidth.
Read →Inference 101The roofline of LLM inference: prefill, decode, TTFT and TPOT
A forward pass costs FLOPs and bytes; prefill does hundreds of FLOPs per byte and hits the compute roof, while decode does a few and is limited by memory bandwidth.
Read →Inference 101PagedAttention: paging the KV cache
PagedAttention stores the KV cache in fixed-size blocks mapped through per-sequence block tables, eliminating reservation and fragmentation waste and enabling copy-on-write sharing.
Read →Inference 101Prefix caching and RadixAttention
Prefix caching keeps the KV cache of earlier requests and reuses it exactly when a new prompt starts with the same tokens, so only the new suffix is prefilled.
Read →Scheduling and decoding
Batching, scheduling and decoding strategies that set throughput and latency.
Chunked prefill (stall-free batching), explained
A scheduling technique that splits prompt processing into fixed-size chunks and batches each chunk with ongoing decodes, bounding the time between tokens.
Read →Inference 101Constrained and structured decoding (grammars, JSON schemas), explained
A decoding method that masks every token that cannot lead to a string in a target grammar or JSON schema, guaranteeing parseable output at a small and avoidable speed cost.
Read →Inference 101Continuous batching (iteration-level scheduling), explained
A serving policy that rebuilds the batch at every forward pass, so finished sequences leave and waiting requests join immediately, keeping memory-bound decode steps full.
Read →Inference 101Medusa and EAGLE: self-speculative decoding heads, explained
Self-speculative methods that attach small trained modules to the target model, proposing several future tokens from its hidden states that the target then verifies in one pass.
Read →Inference 101Prefill/decode disaggregation, explained
A serving architecture that runs prompt processing and token generation on separate GPU pools and transfers the KV cache between them, so each phase gets its own resources and parallelism.
Read →Inference 101Speculative decoding with draft models, explained
A lossless decoding method in which a small draft model proposes several tokens and the target model verifies them in one parallel pass, accepting each with probability min(1, p/q).
Read →Serving post-trained models
Adapters, quantisation and deployment patterns for tuned models.
FP8 and FP4 inference, SmoothQuant and activation quantization
FP8 and FP4 inference quantize activations as well as weights so tensor cores run at low precision, and the whole difficulty is a few activation channels hundreds of times larger than the rest.
Read →Inference 101Tuning for latency SLOs: batch size, token budgets, TTFT, TPOT and goodput
Latency SLO tuning picks the batch cap, per-step token budget and output limits that maximise goodput: the highest request rate at which most requests still meet TTFT and TPOT targets.
Read →Inference 101Merge the LoRA or serve the adapter?
Merging a LoRA gives zero inference overhead and one checkpoint per fine-tune; serving it unmerged costs a little per step but lets many fine-tunes share one deployment.
Read →Inference 101Multi-LoRA serving: S-LoRA, Punica and adapter batching
Multi-LoRA serving keeps one copy of the base model on the GPU and batches requests for many different LoRA adapters together, adding a small per-adapter cost per decode step.
Read →Inference 101Parallelism for serving: tensor, pipeline and expert parallelism
Serving parallelism splits a model across GPUs by matrix (tensor), by layer (pipeline) or by expert, trading lower latency and more memory against communication on every decode step.
Read →Inference 101Weight-only quantization: GPTQ, AWQ and keeping your fine-tune
GPTQ and AWQ store weights in about 4 bits using a little calibration data, cutting decode memory traffic nearly 4x, and the calibration set decides how much of a fine-tune survives.
Read →Bring a model and a metric.
Tell us what the model should get better at. A post-training engineer replies within one business day with a plan, a compute estimate and an eval design.