Modern AI agents handle typical office tasks quite well, but how do you test whether they are ready to work in a complex corporate environment — for example, in telecom? General tests are not enough here: realistic scenarios with actual documents, databases, and atypical user questions are needed. That is exactly why Telco-GAIA was created — a bilingual multimodal benchmark built around data from a real telecom operator. It makes it possible to assess how well an AI agent can reason, use tools, and extract information from different sources.
Why such a benchmark is needed
Most open tests for AI agents resemble puzzles: they are either too synthetic or do not require real work with corporate data. Telco-GAIA closes this gap. Its questions are designed so that the agent encounters tasks familiar to telecom operator employees: finding information on a website, matching it with database records, and clarifying details in archived documents. This is not a test of the model's memory, but a test of its ability to work with multiple sources and draw conclusions.
The benchmark is bilingual: all tasks are formulated in English and Arabic. This choice is not accidental — it reflects the real needs of operators in a region where Arabic is the working language, but a significant portion of internal documentation remains in English.
What is inside Telco-GAIA
The benchmark includes 100 question-answering tasks, each verified by a human. Importantly, the tasks require not a one-word answer, but a series of logical steps — 4.2 steps on average. All questions are divided by modality types: text, tables, and graphics. This means the model must work equally confidently with a PDF document, an image, or the structure of an SQL query.
To solve a task, the agent needs to access three heterogeneous sources:
- A static snapshot of the operator's website — HTML pages, images, and related PDF files.
- A synthetic relational database — its structure mimics the operator's internal system, but the data in it is safe for publication.
- External web archives — additional materials that may be needed to verify facts or clarify context.
How the evaluation works
One of the main principles of Telco-GAIA is objectivity. The authors rejected trendy LLM judges, where one model evaluates another's answers. Instead, normalized exact string matching is used, and the entire benchmark is packaged in an isolated Docker environment. This means the result does not depend on a specific machine or library versions: run the container — get the same numbers. This approach makes evaluation deterministic and reproducible, which is critical for research and comparing different models.
What the tests showed
The developers ran a reference agent with twelve different models — both commercial and open — through the benchmark. The results were modest: even the strongest model handled only 71% of the tasks. If the budget for tool calls is cut to a moderate level, the result drops to about 40%. In other words, the agent starts noticeably "cutting corners": it does not take extra steps, does not check details, and makes more mistakes in non-standard cases.
Models perform especially poorly in categories that require using visual information — images and graphics. The average result there is below 30%. This shows that understanding documents with pictures remains a serious growth area for AI agents. Overall, Telco-GAIA captures a significant gap between expectations for a corporate agent and the real capabilities of modern LLMs.
Conclusions
Telco-GAIA is not just another benchmark for research. It is a practical tool for companies that build their own AI agents and want to understand their limitations in advance. Thanks to its reproducible environment and verified tasks, it can be used as an internal quality standard.
Moreover, the approach embedded in Telco-GAIA can be easily adapted to other closed domains — from banking to logistics. The format provides a ready-made template: take real data, carefully anonymize it, split it into independent sources, and build an environment where the agent must demonstrate its reasoning skills. This is exactly the shift the market needs: from general knowledge tests to checking practical competencies on plausible corporate data.
