Token Splitting in Multimodal Transformers: When Attention Preserves Representational Contribution
This work investigates the conditions under which splitting an input representation into more elements leaves its contribution unchanged, provided that content, position, visible context, and total mass are preserved. For positive attention kernels with a non-decreasing local mass multiplier, invariance is achieved only when the multiplier is linear; on Qwen2.5-Omni-7B and video benchmarks, mass weighting reduces drift and improves accuracy under compression.
Refinement Symmetry: What the Principle Checks
The number of attention weights depends on the number of tokens. Changing the signal representation changes that number. This means that splitting the input affects the mechanics of attention.
The paper “Refinement Symmetry in Multimodal Transformers” (Yuhao Du, Shunian Chen) was submitted on September 26, 2026. It introduces refinement symmetry as a condition for preserving contribution.
Splitting a representation preserves its contribution if four properties are preserved:
- content;
- position;
- visible context;
- total mass.
Practical criterion: check these four properties before changing the tokenization. Losing even one means the contribution is no longer guaranteed.
Linearity of the Local Mass Multiplier
The authors build on proportional attention and quadrature attention. Their conclusion: invariance to splitting requires a linear local mass multiplier. The condition holds for any fixed positive attention kernel. The multiplier is also non-decreasing.
This requirement leads to three checks:
- a nonlinear multiplier changes the contribution when splitting;
- linearity preserves the contribution;
- non-decreasing behavior narrows the range of acceptable schemes.
Practical criterion: check that the weighting scheme is linear before splitting. Nonlinearity predicts a change in the result.
Physical Coupling and Attention Error Bounds
Physical coupling applies to modified representations. It bounds attention error. The bound separates two sources:
- changes in features;
- redistribution of weights.
Practical criterion: analyze both sources separately. This shows what needs fixing—features or weights.
Evaluation on Qwen2.5-Omni-7B: MVBench Answers
In Qwen2.5-Omni-7B, tripling half of the visual tokens changes 255 out of 3,586 answers on MVBench. This is the result with standard attention. Measure-based weighting preserves every answer. Visibility remains comparable.
| Scheme | Changed answers | Visibility |
|---|---|---|
| Standard attention | 255 out of 3,586 | — |
| Measure-based weighting | none | comparable |
Conclusion: the divergence is caused not by splitting itself, but by how mass is accounted for.
Practical criterion: switch to measure-based weighting when splitting visual tokens. It preserves answers with the same visible context.
Frame Resampling and Distribution Drift
With natural frame resampling, measure-based weighting reduces distribution drift. The effect goes beyond preserving individual answers.
Practical criterion: when changing the frame rate, monitor distribution drift, not just accuracy. Measure-based weighting reduces it.
Video Compression: Measured Gains
The evaluation used twofold merging of frozen video encodings. Five random seeds were used. The accuracy gain was positive across all partitions.
| Baseline scheme | Accuracy gain |
|---|---|
| Global weighting by token count (average group mass) | 1.04 percentage points |
| Standard attention | 0.93 percentage points |
The advantage over global weighting by token count is also observed on WorldSense. It depends on the compression budget.
Practical criterion: when using aggressive video compression, check the budget. The gain from measure-based weighting is not constant.
What the Representation Principle Offers
The authors present the result as a representation principle with a measured gain in robustness to partitioning. This is not about one-off fixes, but a rule for handling splitting.
Practical criterion: first fix the representation and its mass, then tune the weights. This prevents splitting from changing the answers.
Frequently asked questions
Related materials
All materials
Moving to the Probability Simplex Reduces the Modality Gap in Contrastive Multimodal Models
The paper arXiv:2610.32890 proposes replacing spherical embeddings with a probabilistic simplex, adding lightweight softmax adapters and Total Variation similarity. Across five retrieval benchmarks, the cross-modal centroid ℓ2 distance decreases by 97–99% while retrieval quality is maintained or improved.
3 November 2026
How ActionRAG Reconsiders Evidence During Diffusion Across the Memory Graph of LLM Agents
A graph-memory architecture is proposed that carefully reconciles entities during indexing and reevaluates relationships at query time based on local structure. On MemoryAgentBench, the method achieved an average accuracy of 55.1, outperforming the strongest retrieval baseline by 5.4 percentage points while keeping synthesis costs low per query.
3 November 2026
Fifteen companies backed MDK Tether for mining automation
Equipment manufacturers, mining pools, and operators have taken on the development of integrations with Tether’s open suite to standardize equipment monitoring and management; the QVAC AI agent is already built on this foundation. Independent confirmation of reduced electricity costs or increased efficiency has not yet been published.
2 November 2026