Why a New Benchmark Is Needed
The more code neural networks write, the more pressing the question becomes: how can we be sure this code actually does what it's supposed to? Standard tests catch some errors but don't provide formal guarantees of correctness. That requires verification—translating a program into the language of rigorous mathematical statements and proving its properties. This work is expensive and complex, so researchers want to offload it to AI.
The problem is that models trained on neat educational examples often get lost when faced with real-world code. Real projects are full of implicit conventions, language quirks, and undocumented behavior. To test how well AI handles such conditions, we need a large-scale proving ground built on real programs, not synthetic exercises.
FVSpec: Real Tests as a Challenge for AI
A group of researchers—Quinn Dougherty, Max von Hippel, Simon Henniger, Hazel Shackleton, and Mike Dodds—presented the FVSpec benchmark. The work appeared on arXiv under number 2606.01008, and in August 2026 the authors released an updated version.
The core idea is to take property-based tests from real Python repositories and turn them into formal specifications in the Lean 4 language. Such tests don't check specific examples but rather function properties: "for any valid input, a certain condition holds." This makes them a natural bridge to mathematical formalization.
The authors collected 11,039 property-based tests from open-source Python projects. Only 2,772 of them—roughly a quarter—could be automatically translated to Lean 4. This yielded 9,415 specifications: about three formalization variants for each successfully translated test. The redundancy is needed because the same property can be written in different ways, and it's not always clear in advance which variant will prove more convenient for subsequent proof work.
Why This Is Hard
Rewriting tests in Lean is not a mechanical syntax substitution. Python and Lean sit in different paradigms: the former features dynamic typing, mutable objects, and imperative constructs, while the latter has a strict system with dependent types. To make a specification match the actual behavior of the code, Python semantics have to be carefully modeled.
An additional challenge is that a property-based test is often written as an imperative script—with loops, exceptions, and state handling. Extracting a clean statement of the form "for any x, P(x) holds" from this is a research problem in itself.
Finally, Lean 4 remains a language with a complex type system that is rarely encountered even in professional development. This is a serious challenge for language models: they need to simultaneously master an uncommon syntax, get used to dependent types, and generate correct definitions.
How the Pipeline and Evaluation Work
For automatic translation, the authors built a pipeline of three LLM agents. Based on the description, the agents divide the work among themselves: one analyzes the source test, another constructs the specification, and a third checks and improves the result. All pipeline code is published, as is the collected data.
Translation quality was assessed along several dimensions. Coverage shows what fraction of real tests can be formalized at all. Separate quality metrics evaluate how well the resulting specification reflects the original behavior of the code. Additionally, the researchers established baselines for proof generation: they ran several automatic and model-based approaches and measured how often they managed to prove the formulated statements. These measurements give future work a reference point for comparison.
What This Means for Practice
FVSpec targets an under-explored area—AI-assisted formal verification of real-world software. Being able to prove properties of educational examples doesn't guarantee success on real code, so a benchmark built on real tests helps fairly compare methods and track progress.
The significance of such an approach grows along with the amount of code generated by neural networks. If AI is increasingly writing programs, then automated correctness-checking tools become not a luxury but a necessity.
Conclusion
FVSpec is a notable step toward making formal verification no longer the domain of narrow specialists but a practical tool. The benchmark combines real property-based tests, the Lean 4 language, and an open LLM pipeline. Thanks to the accessible data and code, any researcher can reproduce the experiments or propose their own approach—giving the community a common point to move forward from.
