People are understandable, LLMs are not: experts failed to explain the logic of models
Researchers compared the hidden factors behind people's and six language models' responses to quantitative reasoning and chemistry tasks. Factors extracted from human responses were successfully interpreted by experts, while most model constructs proved opaque to them.

People are understandable, LLMs are not: experts failed to explain the logic of models
When we look at test results, it seems natural to assume that artificial intelligence solves tasks "like a human": using the same concepts, rules, and reasoning. However, new research calls this assumption into question. Scientists have found that the hidden structures that determine LLM responses are, in most cases, not amenable to human interpretation — even for domain experts.

What exactly was tested
The authors of the paper (arXiv:2608.17810) — Alona Strugatski, Likol Zeinfeld, Jason Cooper and their colleagues — took responses from humans and six different LLMs on quantitative reasoning and chemistry tasks. Using exploratory factor analysis, they identified hidden latent factors for each group that presumably underlie successful performance on these tasks.
Then, subject-matter experts blindly studied the resulting structures and tried to assign pedagogical meaning to the factors: for example, "skill in working with formulas" or "understanding of stoichiometry." The blind format ruled out hints — the specialists did not know whether the graphs referred to humans or to models.
The result: an interpretability gap
Experts had almost no trouble making sense of the factors derived from human responses. Most constructs received a meaningful explanation that could easily be translated into recommendations for teaching or diagnostics.
With LLMs, the picture was different:
- In quantitative reasoning, experts failed to meaningfully interpret any of the factors identified for the models.
- In chemistry, the situation was slightly better, but still far from ideal: only about half of the structures could be explained.

What this means
The authors draw an important conclusion: when LLMs show high results on tests, it does not mean they reason in the way we are used to. Models may find statistically efficient but completely "alien" ways of solving problems. Their internal mechanisms operate at the level of patterns that carry no understandable pedagogical or cognitive meaning.
This observation challenges the common practice of drawing conclusions about the level of knowledge or competencies from AI results. If we cannot explain which specific constructs underlie the responses, we risk attributing non-existent abilities to models — or, conversely, missing real limitations.
Where to go next
The study does not claim that interpreting LLMs is impossible in principle. It only shows that the classical approach, designed for studying human cognition, does not transfer directly to models. It may be necessary to develop fundamentally new methods of analysis — ones that account for the "non-human" nature of latent structures.
For now, the question remains open: if experts cannot explain the logic of models, can we rely on their use in education and knowledge assessment? The answer will likely require a revision of both approaches to evaluating AI and our expectations of machine "intelligence."
Frequently asked questions
Related materials
All materials
Validating Candidates Costs More Than Generating Them: Honeypots Against LLM Vulnerability Hunting
The study examines the cost asymmetry in agentic vulnerability discovery: generating candidate versions is cheap, while confirming them through reachability analysis, execution, and PoC development consumes most of the budget, making verification a potential standalone target for defense. The proposed RedHerring—repository-adapted decoys with a hidden certificate—reduced the number of real vulnerabilities found by 38.7–60.4% at equal cost across 33 OSS-Fuzz projects; even after the agent was warned, the reduction remained at 37.2%.
9 November 2026
Checking LLM Hallucinations via SQL-Based Source Representation Without Fine-Tuning
The study examines whether an LLM can apply a basic symbolic skill—building a SQL database from source documents—to ground its outputs in sources without supervision when detecting hallucinations. On RAGTruth and DiaHalu, this neuro-symbolic pipeline proves more accurate than direct prediction and comparable to state-of-the-art methods, without requiring domain-specific fine-tuning.
9 November 2026
MathAdv: What the Final Accuracy of Machine-Generated Proofs Conceals
A diagnostic suite of tasks spanning 13 undergraduate and graduate mathematics topics evaluates machine-proving systems in Lean 4 not only by their final results, but also by their subject-matter mastery, the quality of their informal reasoning, and their robustness to rephrased conditions. A component-by-component analysis shows that translating a problem into formal form remains the main obstacle, and that success depends heavily on the area of mathematics and how the problem is framed.
9 November 2026