MathAdv: What the Final Accuracy of Machine-Generated Proofs Conceals
A diagnostic suite of tasks spanning 13 undergraduate and graduate mathematics topics evaluates machine-proving systems in Lean 4 not only by their final results, but also by their subject-matter mastery, the quality of their informal reasoning, and their robustness to rephrased conditions. A component-by-component analysis shows that translating a problem into formal form remains the main obstacle, and that success depends heavily on the area of mathematics and how the problem is framed.

What Exactly Does the Benchmark Measure
MathAdv is a diagnostic benchmark for evaluating theorem-proving systems. It covers 13 areas of university- and graduate-level mathematics. The benchmark evaluates theorem proving in Lean 4 and up to three auxiliary tasks.
| Component | What it evaluates |
|---|---|
| Theorem proving in Lean 4 | The system’s ability to construct a formal proof |
| Multiple-choice questions | Mathematical knowledge |
| Fill-in-the-blank tasks | Informal reasoning |
| Expert reformulations of problem statements | Robustness to how a problem is presented |
Overall theorem-proving accuracy does not show at which stage a system falls short. Component-level evaluation separates knowledge, informal reasoning, formalization, and robustness. Practical criterion: look at results by component, not just the final figure.
What Overall Accuracy Doesn’t Reveal About Problem Conditions
Overall accuracy obscures how a system responds to hints and reformulations. A single metric does not distinguish between these conditions. The table shows which effects get lost in the aggregate score.
| Condition | Observation | What to check |
|---|---|---|
| Natural-language hints | Help general-purpose large language models, but can hinder specialized models | Results with and without hints |
| Equivalent reformulations | Reveal significant robustness limitations | Results on reformulated problems |
Natural-language hints help general-purpose large language models. The same hints can hinder models specialized in theorem proving. Mathematically equivalent reformulations reveal significant robustness limitations.

Overall accuracy does not separate these effects from the aggregate theorem-proving result. A single metric does not show where a hint helps and where it hinders. Practical criterion: evaluate sensitivity to hints and robustness to reformulations separately.
Formalization and Differences Across Areas of Mathematics
Formalization remains a serious bottleneck. System results vary substantially across different areas of mathematics. The benchmark covers areas of university- and graduate-level mathematics. Evaluation in Lean 4 separates formal proof from auxiliary tasks.
| Observation | What it means for evaluation |
|---|---|
| Formalization is a bottleneck | Overall accuracy may be limited by translating a problem into a formal environment |
| Differences across areas | The aggregate result obscures differences between areas |
Practical criterion: compare systems by area of mathematics and by the formalization stage. A single overall metric does not show exactly where a system falls short in proving a theorem.
How to Use the Benchmark When Choosing a System
Component-level evaluation reveals models’ capabilities and the reasons for their failures. The dataset and evaluation scripts are available via the link on the paper’s page. The first version of the paper was submitted on 26 Aug 2026; the latest listed version is v3, dated 19 Oct 2026.
Evaluation criteria:
- results broken down by area of mathematics;
- a separate result for formalization in Lean 4;
- comparison of performance with and without natural-language hints;
- evaluation on mathematically equivalent reformulations;
- the paper version and date describing the evaluation.
This set of criteria reveals what overall accuracy obscures. Practical selection criterion: compare the component profile, not just the overall accuracy.
Frequently asked questions
Related materials
All materials
AffordanceWAM: Affordances as an intermediary between scene prediction and manipulator control
The work describes a generative model that jointly predicts future RGB observations, scalar fields, affordance heatmaps, and continuous robot commands, using human videos to broaden interaction diversity without action annotations or retargeting. On RoboCasa, CALVIN ABC→D, and real-world tasks, the approach delivers consistent gains over RGB-only and robot-data-only baselines.
9 November 2026
Presentation Editing as Controlled Operations: The EditPPT Framework and DeckEdit-Bench Benchmark
The authors break down slide editing into specific objects, permitted actions, and checks to ensure unrelated content remains untouched, rather than relying on free-form code generation. To measure quality, they created DeckEdit-Bench (183 tasks across 28 human-made presentations), and the multi-agent EditPPT framework achieves a 66.7–74.9% task success rate across five base models while preserving over 97% of the content.
8 November 2026
Non-Euclidean Factor Geometry: The FactoMap Map for Disentangled Representations
A method is proposed for describing factor space through regions, generator-defined identifications, and local scales, making it possible to distinguish topologically similar cases with different geometries. FactoMap trains interpretable prototypes on a structured grid and, in experiments, demonstrates more robust preservation of factor continuity and separation.
8 November 2026