MAPLE: Transferring the budget of routed experts between layers of an MoE model without modifying weights

20 September 202626 views

The work shows that a uniform number of routed experts across all layers systematically underperforms an uneven allocation, and proposes a pluggable mechanism based on estimating layer sensitivity. On four MoE models, the approach with a 75% expert budget outperforms both the uniform scheme and expert pruning, and in SGLang it reduces serving latency by 32.2% and increases throughput by 47.4%.

MAPLE: Transferring the budget of routed experts between layers of an MoE model without modifying weights

The problem of uniform expert budget

Sparsely activated MoE transformers fix the same number of routed experts across all layers. This ignores the documented heterogeneity of redundancy between layers.

MAPLE is a plug-and-play framework. It redistributes the routed-expert budget across the layers of any pretrained MoE LLM. The method does not change the weights and does not require retraining.

Key properties:

  • Heterogeneous budget allocation across layers.
  • No modification of model weights.
  • No retraining.
  • Applicability to a pretrained MoE LLM.

Practical criterion: the approach is needed when you want to change the expert allocation without modifying weights or retraining.

Layer sensitivity and a closed-form formula

MAPLE probes each layer. The method checks how a layer responds to a change in the number of experts. Sensitivity is quantified with three measures.

Budget transfer stages:

  • Probing each layer.
  • Assessing the response to a change in the number of experts.
  • Quantifying sensitivity with three measures.
  • Deriving the analytically optimal budget allocation.
  • Refining with a genetic search constrained by sensitivity.

The analytical allocation directs capacity to sensitive layers. In redundant layers, the budget reduction is absorbed. The genetic search uses per-layer sensitivity as a prior. This yields faster convergence and better allocation quality.

Conclusion: budget transfer relies on measurable sensitivity rather than a uniform rule.

Comparison with uniform and pruning-based approaches

MAPLE outperforms uniform and pruning-based baselines on four MoE models of different scale and architecture. The comparison is run at a 75% routed-expert budget.

VariantRouted-expert budgetResult
Uniform baseline75%Inferior to MAPLE
Pruning-based baseline75%Inferior to MAPLE
MAPLE75%Outperforms both baselines
Uniform baseline100%Inferior to MAPLE at 75% on DeepSeek-MoE-16B on ARC-E, ARC-C, and BoolQ

On DeepSeek-MoE-16B, MAPLE uses only 75% of the experts. At the same time, it beats the original 100% uniform baseline on ARC-E, ARC-C, and BoolQ.

Accuracy gains:

  • ARC-E: from 65.09 to 71.40.
  • ARC-C: from 48.49 to 51.50.
  • BoolQ: from 80.03 to 82.38.

Conclusion: heterogeneous allocation can be more effective than simply enabling more experts.

Serving efficiency in SGLang

The MAPLE implementation in SGLang reduces end-to-end serving latency on a single GPU by 32.2%. Throughput increases by 47.4%.

The method does not change the weights. Therefore, it can be integrated into an existing MoE serving stack. This makes budget transfer a practical lever for inference.

Practical criterion: for single-GPU serving via SGLang, the method delivers lower latency and higher throughput.

Selection criteria

MAPLE is worth considering in several cases:

  • You need to reduce the routed-expert budget without retraining.
  • The model is a pretrained MoE LLM.
  • You need to account for per-layer heterogeneity rather than a uniform rule.
  • You need to improve MoE inference efficiency.

Conclusion: heterogeneous expert allocation becomes a fundamental and practical axis for improving MoE efficiency.

Frequently asked questions

Related materials

All materials
Canonical Polynomial Invariants for Diagnosing RL Simulator Failures: Where Are They Actually Useful?
News

Canonical Polynomial Invariants for Diagnosing RL Simulator Failures: Where Are They Actually Useful?

The study examines whether incorporating physical laws into learnable dynamics models is justified: invariants extracted from trajectories and reduced to canonical form using Gröbner bases yield modest forecasting gains, but prove powerful for diagnostics, enabling the localization of violated constraints and identification of specific erroneous parameters.

10 November 2026
AURA-Eval: How to Check Whether an LLM Agent Recognized a Risk and Changed Its Next Step
News

AURA-Eval: How to Check Whether an LLM Agent Recognized a Risk and Changed Its Next Step

The framework evaluates LLM agents’ behavior not by their final verdict, but by how they respond to risk at key decision points: tool-call trajectories are interrupted, after which the model must continue reasoning and acting, while a panel of LLM judges assesses risk detection, the nature of the step, and its safety. Across 1,249 tasks drawn from 157 R-Judge trajectories, unsafe actions become more frequent among 20 models when no safe path is available, and removing oversight before execution turns 52.3% of initially safe cases into unsafe ones.

10 November 2026
Chern–Simons Reservoir Memory: Autonomous Connectivity Dynamics and Density Transport
News

Chern–Simons Reservoir Memory: Autonomous Connectivity Dynamics and Density Transport

The article explores how a reservoir preserves the history of input signals both in its density distribution and in its geometric degrees of freedom. A comparison of four mechanisms shows that current creates a separate connected component that alters the subsequent density response, while accounting for all measured features yields a small improvement in decoding pulse order.

10 November 2026
MAPLE: Transferring the budget of routed experts between layers of an MoE model without modifying weights