Model Merging by Compatible Groups: GRAM Reduces Subspace Interference

The Problem of Subspace Interference

Merging different models into one is necessary for multi-task deployment. In regulated fields such as healthcare, data often remains local. As a result, merging can only use fine-tuned weights.

Earlier methods apply the same merging approach to every task. They do not account for geometric relationships between learned subspaces. If task updates occupy misaligned subspaces, projecting them onto a shared low-rank basis discards important directions. The authors call this phenomenon subspace interference.

The projection error for each task is determined by principal-angle alignment. Principal-angle alignment shows how closely a task's subspace matches the shared subspace. The takeaway: uniform merging creates a risk of losing directions that are important for individual tasks.

How GRAM Works: Groups Based on Principal Angles

GRAM (Group-wise Rank-Aware Model Merging) uses the principal-angle geometry of task vectors. The method works in three stages:

1. Calculate the principal angles between task vectors.
2. Use these angles to determine which tasks to merge together.
3. Perform standard SVD-based merging within each group.

At the merging stage, the method uses only fine-tuned weights. It does not require access to the original data. According to the authors, grouping geometrically compatible tasks provably reduces projection error within each group.

Practical criterion: GRAM is suitable when data cannot be shared and tasks may occupy misaligned subspaces. The method provides a way to split tasks into compatible groups before merging.

What Changes at K=2

The scenario of adding just one model (K=2) demonstrates GRAM's effect. In this case, the method significantly reduces subspace interference. It consistently outperforms previous merging methods in tests.

The evaluations cover three areas:

• language understanding — GLUE with Flan-T5;
• computer vision — 8-task CLIP;
• medical instruction tuning — Qwen2-7B-Instruct with four English/Chinese medical LoRAs.

The takeaway: the advantage is already apparent when merging two models. For the minimal scenario of adding one fine-tuned model, GRAM offers a tested way to reduce interference.

Selection Criteria

Choosing between uniform merging and a group-wise approach depends on the task conditions. Below are criteria based on the description of GRAM.

Practical takeaway: if data remains local and tasks differ in their subspace geometry, group-wise merging reduces the risk of losing directions. Uniform merging across all tasks does not account for this risk.