The hidden complexity of SHACL
SHACL is considered one of the key languages for validating RDF graphs, as it allows you to check whether data conforms to given schemas and constraints. But the problem is that creating shapes requires serious technical expertise, which specialists in a particular subject area usually lack. It would be much more convenient to describe the necessary constraints in natural language and get ready-made formal code. This is precisely the task that modern code generation models are trying to solve, but until now it was unclear how to objectively evaluate their results.
NL2SHACL-Bench: a new measurement tool
A team of researchers — Y. Zhou, N. Bobet, and M. Acosta — presented a test suite called NL2SHACL-Bench, designed for translating natural language requirements into SHACL shapes. The work was accepted at the ISWC 2026 conference, and two versions of the preprint have been posted on arXiv: the first appeared in late July 2026, and an updated version in late August. The paper describes in detail the structure of the benchmark and the methodology of the experiments.
Previously, no specialized benchmark for such tasks existed. Researchers had to collect their own examples without a common standard, which made it difficult to compare solutions with each other. The new suite is intended to fill this gap and provide the community with a common point of reference.
Why simple string comparison doesn't work
One of the main challenges in evaluating NL2SHACL systems is semantic equivalence. Two shapes can express the same rule but differ in structure or serialization method. If you only compare the textual representation, results that are identical in meaning will be rejected, while superficially similar ones will be incorrectly accepted. Therefore, the benchmark authors focused on methods that check semantic correspondence rather than simple string matching.
Four modern large language models participated in the experiments. It turned out that they generate syntactically correct SHACL quite confidently, but perform noticeably worse on complex logical and structural patterns that require precise reproduction of the meaning of constraints. This is an important signal: code that looks neat does not always reflect what the user actually meant.
What this changes
The emergence of NL2SHACL-Bench is an important step towards creating practical tools for working with RDF graphs. Now developers and researchers can measure model progress not by eye, but using a unified test suite. This will make it possible to more accurately identify the weaknesses of language models and gradually move closer to a solution that truly understands human intent.
It is too early to say that generating SHACL from text is completely solved — complex constraints still stump the models. But having a standard benchmark makes further research systematic and its results comparable with each other.
