Model merging explained: soups, task arithmetic, TIES, DARE and SLERP

Model merging combines the weights of two or more fine-tuned models into a single model, without further training and without the inference cost of an ensemble. The simplest version averages the parameters of several fine-tunes of the same base (a model soup). More selective versions work on task vectors, the difference between a fine-tuned model and its base: they add them with a scaling factor (task arithmetic), trim small entries and resolve sign disagreements before adding (TIES), randomly drop most entries and rescale the rest (DARE), or interpolate along an arc instead of a straight line (SLERP). In the post-training stack merging sits after training, as a cheap combination step: fold a maths fine-tune and a coding fine-tune into one model, recover general ability lost during a narrow SFT run, or average the checkpoints of a hyperparameter sweep instead of keeping only the best one.

Where the methods come from#

Merging works because fine-tuned models that share a pre-trained initialisation tend to land in the same low-loss region. Frankle et al. (2020) formalised the idea as linear mode connectivity: two networks are linearly connected if every point on the straight line between their weights has low loss. They found that networks trained from different random initialisations generally are not, while copies trained onward from a shared, sufficiently trained starting point are. Fine-tuning from one pre-trained model is the extreme case of a shared starting point, and Wortsman et al. (2022a) observed that such fine-tunes often lie in a single low-error basin. That observation underlies every method here.

Model soups showed the effect at scale: averaging fine-tunes of a ViT-G model from a hyperparameter sweep reached 90.94% top-1 on ImageNet, a state-of-the-art result at the time, at the inference cost of one model. Wortsman et al. (2022b) had earlier used the same interpolation between a zero-shot and a fine-tuned CLIP (WiSE-FT) to recover robustness lost in fine-tuning. Task arithmetic turned averaging into an algebra over skills. TIES and DARE address what goes wrong when you add several task vectors at once: they interfere. SLERP came to LLM merging through community practice and the open-source mergekit library (Goddard et al., 2024), which implements all of these methods behind a single configuration file.

The math, method by method#

Let ${\theta }_0\in {\mathbb{R}}^d$ be the base (pre-trained or instruction-tuned) weights and ${\theta }_1,\dots ,{\theta }_n$ the weights of $n$ models fine-tuned from it. Define task vectors ${\tau }_i={\theta }_i-{\theta }_0$. Every method below outputs a merged ${\theta }_{\text{m}}$.

Uniform and greedy soups

The greedy soup sorts the candidates by validation accuracy and adds each to the running average only if the average's validation accuracy does not drop. Soups are intended for models fine-tuned on the same task with different hyperparameters, where the average lands nearer the centre of a flat basin. Wortsman et al. relate the benefit to the flatness of the loss around the solutions. A simple way to see it: if the loss is roughly convex along the line between two models, Jensen's inequality gives their average a loss no higher than the mean of their losses.

Task arithmetic

With $\lambda =1/n$ this is the uniform soup. Larger $\lambda$ keeps more of each task's update; Ilharco et al. choose $\lambda$ on held-out validation data. Negation, ${\theta }_0-\lambda \tau$, removes a behaviour (they use it to reduce toxic generation), and analogies of the form ${\tau }_D\approx {\tau }_C+({\tau }_B-{\tau }_A)$ can improve a task with no data for it.

TIES-Merging

Adding task vectors fails in two ways: many entries are small, redundant changes that add noise, and where two tasks moved a parameter in opposite directions their sum cancels both. TIES runs three steps per parameter $j$:

The elected sign is the sign of the summed trimmed values, so it is decided by total magnitude rather than by vote count. The disjoint mean averages only over models that agree, so an entry kept by one model is not diluted by zeros from the others. The paper's default is $k=20\%$ and $\lambda =1$; a search over $k\in \{10,20,30\}$ and $\lambda \in [0.8,3.0]$ found $k=20$ best and $\lambda$ between 0.9 and 1.1 equivalent.

DARE

DARE ("Drop And REscale") is a preprocessing step, not a merge rule: it sparsifies each task vector at random while keeping its expected value, and the result is fed to task arithmetic or TIES (dare_linear and dare_ties in mergekit). The claim behind it is that SFT deltas of large language models are small (the paper reports values typically within 0.002) and extremely redundant, so 90% or even 99% of them can be dropped without hurting the fine-tuned model. Sparser task vectors overlap less, so they interfere less when summed.

SLERP

For two weight vectors $\mathbf{a},\mathbf{b}$ with angle $\Omega =\arccos \left(\frac{\mathbf{a}\cdot \mathbf{b}}{\Vert \mathbf{a}\Vert \Vert \mathbf{b}\Vert }\right)$,

For unit vectors the linear midpoint has norm $\cos (\Omega /2)$, shorter than either endpoint; SLERP stays on the arc and keeps the norm. In practice mergekit applies SLERP per tensor (with $t$ optionally varying by layer and by tensor type) and only between two models. Its variant nuslerp can instead interpolate task vectors relative to a base.

Worked example: three task vectors by hand#

Take three task vectors over five parameters:

Trim. Keeping the three largest magnitudes in each vector zeroes ${\tau }_1$'s p2 and p4, ${\tau }_2$'s p2 and p4, and ${\tau }_3$'s p4 and p5. Elect. The sums of trimmed values are $(0.8,0.3,-0.1,0,-0.7)$, so the elected signs are $(+,+,-,0,-)$. Disjoint mean. For p1, the agreeing entries are 0.5 and 0.4, mean 0.45; ${\tau }_3$'s $-0.1$ is discarded. For p2, only ${\tau }_3$'s 0.3 survived trimming, so it is kept at full strength rather than divided by three. For p3, the values 0.3, $-0.6$ and 0.2 sum to $-0.1$, so the single large negative entry wins and the merged value is $-0.6$, where task arithmetic averages to $-0.033$. That last parameter shows TIES's bias in both directions: it protects a large, confident change from being cancelled, and it also discards two smaller changes that together pointed the other way.

DARE on ${\tau }_1$ with $p=0.5$ and seeded mask $(1,0,0,0,1)$ gives $(1.0,0,0,0,-0.8)$: two survivors, each doubled. The expected value equals ${\tau }_1$, but a single draw does not, which is why DARE relies on the very high redundancy of large-model deltas. SLERP: for two unit vectors at 90 degrees, the linear midpoint has norm $\cos 45^{\circ }=0.707$ while SLERP's has norm 1.0; at 30 degrees the gap is 0.966 versus 1.0; at 10 degrees, 0.996 versus 1.0.

Toy run: merging two fine-tunes of a small network#

To compare the methods end to end, we pre-trained a small MLP (16 inputs, two tanh hidden layers of 64, 6 classes) on a broad classification task defined by a random teacher network, then fine-tuned it separately on two related tasks, A and B. Each task uses a perturbed copy of the teacher and inputs drawn from its own region of input space, so the tasks share structure with pre-training but disagree in places. Fine-tuning used Adam for 600 steps; every merge coefficient was chosen on a validation split and evaluated on a separate test split; 5 seeds. All numbers are the script's output.

Several results are worth reading carefully. Task arithmetic with $\lambda$ tuned on validation (it chose 0.7 to 1.0) beat the plain average by 6 points, because $\lambda =0.5$ halves each task's update. SLERP matched the average to the reported precision: the full weight vectors of the two fine-tunes had cosine similarity of about 0.987, so the arc and the chord are almost the same path (the linear midpoint's norm was 0.997 of SLERP's). In this small, dense network TIES's default 20% trim hurt: it fell below the plain average, and accuracy recovered as $k$ grew. The two task vectors disagreed in sign on about 47% of entries, yet electing signs without trimming still did not beat tuned task arithmetic. DARE's random dropping cost about 3 points against task arithmetic.

The reason is the premise behind TIES and DARE: redundancy. Here the fine-tuning deltas were 10 to 13% of the base weights' norm, and dropping 90% of a single model's delta took task A accuracy from 75.2% to 68.8% (99% dropped: 47.3%, near the base). The DARE paper reports that LLM SFT deltas are far smaller and more redundant, so the same operations remove noise rather than signal. The lesson for practice is not that TIES and DARE do not work. It is that their defaults encode an assumption about the models you are merging, and you should check it, for example by sparsifying one model alone and measuring what it loses, before trusting the defaults.

The procedure, step by step#

base = load("base-model")                  # common ancestor; must match architecture and tokenizer
ckpts = [load(p) for p in candidate_paths] # fine-tunes of that base
taus = [{k: c[k] - base[k] for k in base} for c in ckpts]
for method, grid in [("task_arithmetic", lambdas), ("ties", ties_grid), ("dare_ties", dare_grid)]:
    for params in grid:
        merged = merge(base, taus, method, **params)          # per tensor, in float32
        score[method, params] = evaluate(merged, validation_suites)
best = argmax(score)                        # pick on validation, report on held-out test
save(merge(base, taus, *best), "merged")

What you need. Two or more checkpoints with identical architecture, fine-tuned from the same base (or from one another). Same tokenizer and vocabulary, or an explicit plan for embeddings. A validation set for every capability you want to keep, because merge coefficients are hyperparameters and the only way to choose them is to evaluate. No training data is needed for the merge itself. Adapters. LoRA adapters trained on the same base can be merged by first materialising each $BA$ product as a dense delta; averaging the $A$ and $B$ factors separately is not equivalent, since $\bar{B}\bar{A}\ne \overline{BA}$.

Where merging fits in a post-training pipeline#

Merging is most useful as a cheap step between expensive ones. Four patterns cover most of its use in practice.

Recovering what a narrow fine-tune broke. A model fine-tuned hard on one domain often loses general instruction following or robustness. Interpolating part of the way back toward the starting checkpoint, ${\theta }_0+t\tau$ with $t$ between 0.5 and 0.9, frequently keeps most of the domain gain and much of the general ability. This is WiSE-FT applied to language models, and it is the first thing to try when a fine-tune regresses on a broad evaluation.

Combining specialists trained in parallel. Teams that train separate maths, coding and tool-use fine-tunes from one base can merge them instead of rebuilding a multi-task data mix. Expect each skill to lose a few points relative to its specialist; whether that is acceptable depends on how the model will be served. If skills can be routed at inference time, separate adapters avoid the loss entirely.

Averaging checkpoints of one run. Averaging the last few checkpoints of an SFT or preference run, or several seeds of the same recipe, is a uniform soup. It usually smooths out the noise of picking a single checkpoint and costs nothing but evaluation.

Merging after alignment stages. Merging a preference-tuned or RL-tuned model back toward its SFT parent trades some of the alignment gain for less drift, which can be a useful knob when the later stage over-optimised. Treat the interpolation coefficient like a KL budget chosen after the fact, and evaluate safety and the targeted behaviour at each setting.

Hyperparameters#

Compute and memory#

Merging needs no gradients and no GPU. It is a streaming element-wise operation over tensors, so memory is a few tensors at a time if done lazily, and time is dominated by reading and writing checkpoints. For a 7B model in bf16 (about 14 GB per checkpoint), merging three models plus a base reads roughly 56 GB and writes 14 GB, minutes on a CPU. Do arithmetic in float32 even for bf16 inputs; small deltas lose precision when subtracted in bf16. The real cost is evaluation: every coefficient setting is a new model to benchmark, so a grid of 20 settings across 5 evaluation suites is 100 evaluation runs. That still compares favourably with multi-task fine-tuning, which needs all the training data in one place and a full training run.

Failure modes and how to detect them#

No shared basin. Models fine-tuned from different bases, or trained long enough to leave the base's basin (continued pre-training on large corpora, long RL runs), may not be linearly connected. The merged model's loss on a simple held-out set will be higher than both endpoints'. Check by evaluating a few points on the straight line between two models before merging; a barrier in the middle means do not merge.

Interference. Each skill degrades a little in the merge, and with many models a lot. Track every source model's own benchmark in the merged model, not only an aggregate. Sign-conflict rate between task vectors is a useful cheap diagnostic.

Format and template conflicts. Merging an instruction model that uses one chat template with one that uses another produces a model that follows neither reliably. Align templates and special tokens before merging.

Scale miscalibration. $\lambda$ too large over-amplifies task vectors and produces degenerate outputs (repetition, broken formatting); too small and the merge regresses to the base. Always sweep it.

Benchmark overfitting. Merges are cheap, so it is easy to try hundreds and pick the best on a public leaderboard. Choose coefficients on validation data that is disjoint from the data you report.

Safety regression. Merging a capability fine-tune into an aligned model can dilute refusal behaviour. Include safety evaluations in the selection suite.

When to merge, and which method#

Implementation#

The core operations in PyTorch, per tensor:

import torch

def task_arithmetic(base, tunes, lam):
    return {k: base[k] + lam * sum(t[k] - base[k] for t in tunes) for k in base}

def ties(base, tunes, density=0.2, lam=1.0):
    out = {}
    for k in base:
        taus = torch.stack([(t[k] - base[k]).float().flatten() for t in tunes])
        n_keep = max(1, int(density * taus.shape[1]))
        thr = taus.abs().topk(n_keep, dim=1).values[:, -1:]          # per-model threshold
        taus = torch.where(taus.abs() >= thr, taus, torch.zeros_like(taus))
        sign = torch.sign(taus.sum(0))                                # elect by total mass
        agree = (torch.sign(taus) == sign) & (taus != 0)
        merged = (taus * agree).sum(0) / agree.sum(0).clamp(min=1)    # disjoint mean
        out[k] = (base[k].float() + lam * merged.view_as(base[k])).to(base[k].dtype)
    return out

def dare(tau, p=0.9, generator=None):
    keep = torch.rand(tau.shape, generator=generator, device=tau.device) >= p
    return tau * keep / (1 - p)

def slerp(a, b, t, eps=1e-8):
    a32, b32 = a.float().flatten(), b.float().flatten()
    cos = torch.dot(a32, b32) / (a32.norm() * b32.norm() + eps)
    omega = torch.arccos(cos.clamp(-1, 1))
    if omega < 1e-4:
        out = (1 - t) * a32 + t * b32
    else:
        out = (torch.sin((1 - t) * omega) * a32 + torch.sin(t * omega) * b32) / torch.sin(omega)
    return out.view_as(a).to(a.dtype)

For real checkpoints, the open-source mergekit library handles sharded weights, lazy loading, tokenizer alignment and layer-wise schedules. A TIES merge is a short YAML file passed to mergekit-yaml config.yml ./merged:

# config.yml
merge_method: ties
base_model: org/base-7b
models:
  - model: org/base-7b-math
    parameters: {density: 0.2, weight: 0.5}
  - model: org/base-7b-code
    parameters: {density: 0.2, weight: 0.5}
dtype: bfloat16

Switch merge_method to task_arithmetic, dare_ties or slerp (exactly two models, with a global t) to compare methods on the same validation suite. In POFE, merging is treated as a post-training step like any other: candidates and coefficients are chosen on validation suites and the merged checkpoint is evaluated before release.

Further reading#

Frankle et al. (2020) for why merging can work at all; Wortsman et al. (2022a, 2022b) for soups and robust interpolation; Ilharco et al. (2023) for task vectors; Yadav et al. (2023) and Yu et al. (2024) for interference-aware merging; Matena and Raffel (2022) for Fisher-weighted averaging, which weights each parameter by an estimate of its importance; Goddard et al. (2024) for the tooling most LLM merges use.

Frequently asked questions#

References#

1. Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, Ludwig Schmidt (2022). Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. arxiv.org/abs/2203.05482
2. Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo-Lopes, Hannaneh Hajishirzi, et al. (2022). Robust fine-tuning of zero-shot models. arxiv.org/abs/2109.01903
3. Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, Ali Farhadi (2023). Editing Models with Task Arithmetic. arxiv.org/abs/2212.04089
4. Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, Mohit Bansal (2023). TIES-Merging: Resolving Interference When Merging Models. arxiv.org/abs/2306.01708
5. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, Yongbin Li (2024). Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch. arxiv.org/abs/2311.03099
6. Ken Shoemake (1985). Animating rotation with quaternion curves. doi.org/10.1145/325165.325242
7. Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, Michael Carbin (2020). Linear Mode Connectivity and the Lottery Ticket Hypothesis. arxiv.org/abs/1912.05671
8. Michael Matena, Colin Raffel (2022). Merging Models with Fisher-Weighted Averaging. arxiv.org/abs/2111.09832
9. Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, Jacob Solawetz (2024). Arcee's MergeKit: A Toolkit for Merging Large Language Models. arxiv.org/abs/2403.13257

Related reading