OpenCollab: programmable agent coordination, controlled execution environment, and policy compliance metric

11 November 20264 views

The framework brings together the specification of agent interaction structures, a shared environment for experiments, and detailed event tracing, while the newly introduced Adherence metric reflects how closely actual execution follows the declared schema. In benchmarks, the Duo configuration achieves a new state of the art, while the single-agent variant uses the fewest tokens.

OpenCollab: programmable agent coordination, controlled execution environment, and policy compliance metric

Purpose of the Framework

OpenCollab is a framework for multi-agent development. It provides unified infrastructure for programmable interaction and a controlled execution environment.

Key elements:

  • multi-agent development;
  • programmable agent interaction;
  • controlled execution environment.

Practical takeaway: the framework is useful when agent coordination needs to be defined programmatically and run in a controlled setting.

Agent Organization as a Design Object

OpenCollab combines the design of agent organization with execution. Organization becomes part of the configuration rather than an external description.

What the design covers:

  • the composition of the agent organization;
  • the specified interaction structure;
  • the connection between structure and execution.

Criterion: the organization must be described in a way that makes it reproducible. Otherwise, the design remains a declaration.

Controlled Execution Environment and Observability

OpenCollab provides experimental control in a shared execution environment. The environment keeps agents within a single setup.

What the environment provides:

  • shared execution conditions for the configurations being compared;
  • experimental control;
  • detailed event streams.

Practical criterion: without a shared environment and event streams, comparing configurations is not reproducible.

Adherence: A Metric for Protocol Compliance

The authors of OpenCollab introduce Adherence. This metric measures how closely the declared agent organization is implemented in practice.

What the metric captures:

  • the declared agent organization;
  • actual execution;
  • the gap between them as a percentage.

In experiments, changing any single configuration dimension shifted Adherence. The metric ranged from 47.2% to 97.2%.

Practical criterion: Adherence shows the gap between the declared organization and actual execution. The metric is evaluated alongside the final results.

Results on Agentic Development Benchmarks

On agentic development benchmarks, OpenCollab (Duo) achieved a new SOTA. The authors compared it with Mini-SWE-agent, OpenHands, Codex CLI, and Claude Code.

According to the authors’ results, a well-designed agent organization can outperform strong existing harnesses. The single-agent OpenCollab configuration used the fewest tokens across all evaluated test suites.

ParticipantResult
OpenCollab (Duo)New SOTA on agentic development benchmarks
Mini-SWE-agentIncluded in the comparison
OpenHandsIncluded in the comparison
Codex CLIIncluded in the comparison
Claude CodeIncluded in the comparison
Single-agent OpenCollab configurationFewest tokens across all evaluated test suites

Selection criterion: compare not only the top SOTA result, but also the token usage of the specific configuration.

Practical Selection Criterion

Start by checking whether agent coordination can be defined programmatically. Then check whether the framework measures adherence to the declared organization. OpenCollab provides Adherence and a shared execution environment for this purpose.

Selection checklist:

  1. Programmable agent coordination.
  2. Availability of the Adherence metric.
  3. Shared execution environment and event streams.
  4. Comparable results on agentic development benchmarks.
  5. Token usage of the selected configuration.

The framework is a good fit if all five points are verifiable. If any point cannot be verified, the choice lacks a sound basis.

Frequently asked questions

Related materials

All materials
OpenCollab: programmable agent coordination, controlled execution environment, and policy compliance metric