The Essence of the Experiment
The FIFA World Cup is an ideal testing ground for AI: only 104 matches, strict rules, and an enormous number of factors. Researchers Jonaid Shianifar and Iias Faiud decided to use this event to create the AI World Cup 2026 benchmark. Ten assistants based on large language models took part in the experiment. Each of them had to make a single prediction for the entire tournament — before the starting whistle.
The conditions were absolutely identical for everyone: the same snapshot of tournament information, a single prompt, the same JSON response schema, and a common scoring procedure. This approach eliminates random differences and allows comparing the capabilities of the models themselves rather than the quality of prompts. The prediction included not only the score of every group-stage match, but also the final group standings, the full playoff bracket, the teams' final positions, as well as the model's confidence level and a brief explanation of its decision.
How scoring worked
The key feature of the AI World Cup is the holistic evaluation of a prediction. Points were awarded for each part of the tournament: for a correctly guessed match score, for the correct position of a team in its group, for the correct winner of a playoff pair, and so on. The authors deliberately made the scoring transparent: the formulas and code are published, so anyone can reproduce the results.
This design is important because it reveals which part of the tournament contributes most to the final ranking. As subsequent calculations showed, that contribution turned out to be extremely uneven. Models were not allowed to revise their answers after the matches started — the prediction was locked in before the tournament. This simulates the real task of an analyst who must deliver a final forecast in advance, without the ability to peek at intermediate results.
Who won the overall standings
First place by a wide margin went to GPT-5.5 Thinking — 744 points. The second result was posted by GPT-5.5 (717 points), third was Gemini (699), and fourth — Qwen 3.7 (687). Interestingly, only the winner predicted Spain's title — in the actual final, the Spaniards beat Argentina 1:0. It seems the model's capacity for long reasoning gave it an edge in such a multi-step task.
Why the playoffs decide everything
The most unexpected finding is the correlation analysis. A model's total score almost perfectly correlated with the points earned for playoff predictions: the correlation coefficient was 0.986. At the same time, the relationship with group-stage results was virtually zero (r=0.055), and with group-position predictions it was even negative (r=−0.103). The cumulative score before the playoffs also had no effect on the final placement (r=−0.054).
This means that success in the benchmark depended not on how accurately a model predicted the scores of individual matches, but on whether it could correctly build the tournament bracket. Mistakes in the group stage could be compensated by accurate playoff forecasts, and vice versa — any misses in the playoffs were fatal to the ranking.
Accuracy on individual matches is not what matters most
The authors separately examined the accuracy of group-stage outcome predictions. Here the best model was Claude Sonnet 4.6 — it correctly guessed 63.89% of outcomes. However, it finished only sixth in the overall standings. So the ability to guess individual matches does not guarantee success in a holistic tournament prediction. A model can excel at local tasks while losing the big picture — for example, misplacing teams in the playoff bracket or making mistakes in final positions.
Confidence is not linked to accuracy
Another important conclusion concerns confidence calibration. The models indicated how confident they were in their predictions, but these estimates had nothing to do with actual results. The correlation of average confidence with outcome accuracy was −0.060, and with the total score — −0.067. In other words, the most confident model was not the most accurate, and the most modest one was not necessarily wrong. This is a reminder that a neural network's self-assessment is not a quality metric but merely a reflection of its internal probability distribution.
What this means for AI forecasting
The authors' main conclusion is that forecasting an entire tournament tests different abilities than predicting matches one by one. The final ranking depends heavily on the design of the scoring system: if points were awarded differently, the order of the models could change. This is an important reminder for anyone using AI in analytics: you need to clearly understand what task the model is solving and what exactly we want to measure.
From a practical standpoint, the AI World Cup 2026 showed that modern LLMs are already capable of building complex multi-stage forecasts, but they are still far from being a reliable "football oracle." Nevertheless, the openness of the materials — raw model responses, scoring code, prompts — makes this benchmark useful for further research. Any team can take it as a foundation and propose its own evaluation scheme. The paper itself on arXiv runs 18 pages, includes 8 figures and 7 tables, and the project repository is listed in the text — so if you wish, you can study all the details down to each model's individual responses.
