19 materials
Five well-known models — from GPT-4o to Llama-3.1 — reach different verdicts on the same text when it is written in Urdu rather than in English translation: the discrepancy reaches up to a third of cases, and some clearly hostile content remains unlabeled. An analysis of nine years of WOAH publications showed that not a single dedicated paper there has been devoted to this language.
Parker Fawcett's rebuild-dossier tool first locks down the app's actual interface — exactly what goes in and what must come out — and only then allows code to be written, guiding the build step by step. Experiments revealed something unpleasant: an agent that honestly followed the rules failed the delayed check, while a rule-breaker passed everything — meaning a successfully passed test suite doesn't certify correctness if the tests themselves can be gamed.
A new arXiv preprint tests whether a language model's self-assessment can be trusted at the moment it chooses a move. In a chess variant with a hidden "royal" status, the stated probability of ≥0.5 matched the piece's actual position only once out of 62 attempts, and nearly the entire calibration deficit came from precisely these episodes.
LLM-based agents can write code inspired by scientific publications, but they often produce implementations that only superficially resemble the original. The SA-Bench diagnostic benchmark breaks down 30 papers from ICLR, ICML, and NeurIPS 2025 into atomic implementation claims and shows how far generated repositories drift from the authors' intent.
Multimodal "omni-judges" evaluate images, video, and speech, but conventional benchmarks are skewed toward successful examples and therefore hide failures. A new balanced D3-Omni suite separates failure types along distinct axes and shows that models readily confirm a satisfied requirement but are far worse at catching a violated one.
A new arXiv preprint 2608.24069 examines a threat that doesn't require access to a system's internals — it's enough to tweak the input data or prompts that agents read. Running four roles (analyst, researcher, trader, risk manager) and four topologies of connections between them across five assets, the authors got the same result every time: no robust configuration was found.
Researchers found that the hidden states of agentic LLMs contain a signal, even before generation, that predicts susceptibility to indirect prompt injections with an AUROC above 0.90. On this basis, a defense was proposed that reduces attack success from 34.6% to zero on one of the models, and a gap was revealed between a model's "knowledge" and its actions.
Researchers found that standard LLM unlearning methods are easily broken by fine-tuning on a small number of forgotten examples. Their proposed Margin Calibration technique adds a non-saturating margin penalty and significantly reduces the success of such attacks across various models and datasets.
New research shows that a biased LLM judge silently breaks the mechanism for discarding failed skills: when the share of false successes exceeds 0.45, the system stops cleaning the skill library, and this is invisible in aggregated metrics. The authors propose a low-cost audit with defect injection to vet the judge before deployment.
