MathAdv: What the Final Accuracy of Machine-Generated Proofs Conceals

9 November 202614 views

A diagnostic suite of tasks spanning 13 undergraduate and graduate mathematics topics evaluates machine-proving systems in Lean 4 not only by their final results, but also by their subject-matter mastery, the quality of their informal reasoning, and their robustness to rephrased conditions. A component-by-component analysis shows that translating a problem into formal form remains the main obstacle, and that success depends heavily on the area of mathematics and how the problem is framed.

MathAdv: What the Final Accuracy of Machine-Generated Proofs Conceals

What Exactly Does the Benchmark Measure

MathAdv is a diagnostic benchmark for evaluating theorem-proving systems. It covers 13 areas of university- and graduate-level mathematics. The benchmark evaluates theorem proving in Lean 4 and up to three auxiliary tasks.

ComponentWhat it evaluates
Theorem proving in Lean 4The system’s ability to construct a formal proof
Multiple-choice questionsMathematical knowledge
Fill-in-the-blank tasksInformal reasoning
Expert reformulations of problem statementsRobustness to how a problem is presented

Overall theorem-proving accuracy does not show at which stage a system falls short. Component-level evaluation separates knowledge, informal reasoning, formalization, and robustness. Practical criterion: look at results by component, not just the final figure.

What Overall Accuracy Doesn’t Reveal About Problem Conditions

Overall accuracy obscures how a system responds to hints and reformulations. A single metric does not distinguish between these conditions. The table shows which effects get lost in the aggregate score.

ConditionObservationWhat to check
Natural-language hintsHelp general-purpose large language models, but can hinder specialized modelsResults with and without hints
Equivalent reformulationsReveal significant robustness limitationsResults on reformulated problems

Natural-language hints help general-purpose large language models. The same hints can hinder models specialized in theorem proving. Mathematically equivalent reformulations reveal significant robustness limitations.

Overall accuracy does not separate these effects from the aggregate theorem-proving result. A single metric does not show where a hint helps and where it hinders. Practical criterion: evaluate sensitivity to hints and robustness to reformulations separately.

Formalization and Differences Across Areas of Mathematics

Formalization remains a serious bottleneck. System results vary substantially across different areas of mathematics. The benchmark covers areas of university- and graduate-level mathematics. Evaluation in Lean 4 separates formal proof from auxiliary tasks.

ObservationWhat it means for evaluation
Formalization is a bottleneckOverall accuracy may be limited by translating a problem into a formal environment
Differences across areasThe aggregate result obscures differences between areas

Practical criterion: compare systems by area of mathematics and by the formalization stage. A single overall metric does not show exactly where a system falls short in proving a theorem.

How to Use the Benchmark When Choosing a System

Component-level evaluation reveals models’ capabilities and the reasons for their failures. The dataset and evaluation scripts are available via the link on the paper’s page. The first version of the paper was submitted on 26 Aug 2026; the latest listed version is v3, dated 19 Oct 2026.

Evaluation criteria:

  • results broken down by area of mathematics;
  • a separate result for formalization in Lean 4;
  • comparison of performance with and without natural-language hints;
  • evaluation on mathematically equivalent reformulations;
  • the paper version and date describing the evaluation.

This set of criteria reveals what overall accuracy obscures. Practical selection criterion: compare the component profile, not just the overall accuracy.

Frequently asked questions

Related materials

All materials
AffordanceWAM: Affordances as an intermediary between scene prediction and manipulator control
Research

AffordanceWAM: Affordances as an intermediary between scene prediction and manipulator control

The work describes a generative model that jointly predicts future RGB observations, scalar fields, affordance heatmaps, and continuous robot commands, using human videos to broaden interaction diversity without action annotations or retargeting. On RoboCasa, CALVIN ABC→D, and real-world tasks, the approach delivers consistent gains over RGB-only and robot-data-only baselines.

9 November 2026
Presentation Editing as Controlled Operations: The EditPPT Framework and DeckEdit-Bench Benchmark
News

Presentation Editing as Controlled Operations: The EditPPT Framework and DeckEdit-Bench Benchmark

The authors break down slide editing into specific objects, permitted actions, and checks to ensure unrelated content remains untouched, rather than relying on free-form code generation. To measure quality, they created DeckEdit-Bench (183 tasks across 28 human-made presentations), and the multi-agent EditPPT framework achieves a 66.7–74.9% task success rate across five base models while preserving over 97% of the content.

8 November 2026
Non-Euclidean Factor Geometry: The FactoMap Map for Disentangled Representations
News

Non-Euclidean Factor Geometry: The FactoMap Map for Disentangled Representations

A method is proposed for describing factor space through regions, generator-defined identifications, and local scales, making it possible to distinguish topologically similar cases with different geometries. FactoMap trains interpretable prototypes on a structured grid and, in experiments, demonstrates more robust preservation of factor continuity and separation.

8 November 2026
MathAdv: What the Final Accuracy of Machine-Generated Proofs Conceals