People are understandable, LLMs are not: experts failed to explain the logic of models
When we look at test results, it seems natural to assume that artificial intelligence solves tasks "like a human": using the same concepts, rules, and reasoning. However, new research calls this assumption into question. Scientists have found that the hidden structures that determine LLM responses are, in most cases, not amenable to human interpretation — even for domain experts.
What exactly was tested
The authors of the paper (arXiv:2608.17810) — Alona Strugatski, Likol Zeinfeld, Jason Cooper and their colleagues — took responses from humans and six different LLMs on quantitative reasoning and chemistry tasks. Using exploratory factor analysis, they identified hidden latent factors for each group that presumably underlie successful performance on these tasks.
Then, subject-matter experts blindly studied the resulting structures and tried to assign pedagogical meaning to the factors: for example, "skill in working with formulas" or "understanding of stoichiometry." The blind format ruled out hints — the specialists did not know whether the graphs referred to humans or to models.
The result: an interpretability gap
Experts had almost no trouble making sense of the factors derived from human responses. Most constructs received a meaningful explanation that could easily be translated into recommendations for teaching or diagnostics.
With LLMs, the picture was different:
- In quantitative reasoning, experts failed to meaningfully interpret any of the factors identified for the models.
- In chemistry, the situation was slightly better, but still far from ideal: only about half of the structures could be explained.
What this means
The authors draw an important conclusion: when LLMs show high results on tests, it does not mean they reason in the way we are used to. Models may find statistically efficient but completely "alien" ways of solving problems. Their internal mechanisms operate at the level of patterns that carry no understandable pedagogical or cognitive meaning.
This observation challenges the common practice of drawing conclusions about the level of knowledge or competencies from AI results. If we cannot explain which specific constructs underlie the responses, we risk attributing non-existent abilities to models — or, conversely, missing real limitations.
Where to go next
The study does not claim that interpreting LLMs is impossible in principle. It only shows that the classical approach, designed for studying human cognition, does not transfer directly to models. It may be necessary to develop fundamentally new methods of analysis — ones that account for the "non-human" nature of latent structures.
For now, the question remains open: if experts cannot explain the logic of models, can we rely on their use in education and knowledge assessment? The answer will likely require a revision of both approaches to evaluating AI and our expectations of machine "intelligence."
