What Is ShowTellArena
ShowTellArena is a benchmark protocol and public dataset. It evaluates understanding of business processes after demonstrations with commentary. Release v1.0 includes 50 business process tasks. The dataset includes recordings, screenshots, narration, fixture seeds, and 502 questions. The tasks cover finance, hiring, procurement, customer decisions, inventory, and logistics.
Takeaway: the benchmark tests retention of a specific process, not general business knowledge.
How the Protocol Works
The business process scenario and test are fixed. Each product is allowed to conduct training through its own interface. This approach ensures that all participants get the same questions. However, training conditions depend on the product interface.
The questions test four skill groups.
| Assessment group | What is assessed |
|---|---|
| Operational rules | Understanding the sequence of steps in a process |
| Boundaries | Understanding the limits of a rule's applicability |
| Exceptions | Handling unusual cases |
| Automation errors | Finding incorrect steps in proposed scenarios |
Takeaway: the protocol separates assessment content from the training method, but does not equalize product interfaces.
What's in the Dataset
Release v1.0 contains 50 tasks. Each task describes a business process. The dataset includes recordings, screenshots, narration, fixture seeds, and 502 questions. The tasks cover six areas: finance, hiring, procurement, customer decisions, inventory, and logistics.
Takeaway: the dataset provides a ready-made foundation for assessing process understanding across different departments.
What the Pilot Found
The article analyzes 218 selected pilot attempts. They cover 39 workflow scenarios. 28 scenarios were completed by all three evaluated systems. The results reveal errors in responses and failures to complete training. The selected pilot is not a controlled comparison of products.
Takeaway: the pilot shows specific failures, but does not provide an overall ranking of systems.
Limitations
The authors describe gaps in dataset validation. They note uneven pilot coverage, exclusions, and the provenance of the evaluations. The pilot is not a controlled comparison. These factors affect how the results should be interpreted.
Takeaway: the results cannot be generalized to all products, processes, or training conditions.
Practical Selection Criterion
When selecting an AI tool for business process automation, it is useful to look at how it handles boundaries and exceptions. These question groups show whether the system understands the limits of rules. The ability to complete training is another indicator. If the system does not complete training, this limits its use. It is also important to keep in mind that the pilot is not a controlled comparison.
Takeaway: ShowTellArena helps assess process understanding, but does not replace testing the tool yourself.
