Clinical rules instead of markup: a new method trains an ophthalmological triage agent on 3,000 dialogues

8 September 202632 views

The authors presented the GAO approach: recommendations from the American Academy of Ophthalmology are compiled into a table of 70 rules, which serves as the sole source of control for training. A fine-tuned 9B model improved alignment with the reference from 61.7% to 74.1%, and recognition of unforeseen cases from 9.5% to 69.0%, with manual annotation used only for evaluation.

Clinical rules instead of markup: a new method trains an ophthalmological triage agent on 3,000 dialogues

Why Conventional Annotation Hinders Training Medical Agents

You cannot train a dialog agent for ophthalmological triage simply on words: you need to show it thousands of examples of conversations with correct answers. In conventional products, such annotation is done by human annotators. In a clinical setting, it is more complicated: annotation requires the involvement of physicians, is expensive, and transcripts of conversations with patients are protected by medical confidentiality. They cannot be handed over to external contractors, which makes scaling supervision practically impossible.

The authors of a recent study proposed moving away from this model. Instead of hiring experts and manually annotating every utterance, they used clinical guidelines themselves as a source of training signal. In ophthalmology, triage rules are strict enough to be reduced to formal operations that can annotate dialogs automatically. This approach not only reduces the cost of the process but also allows sensitive data to remain within the research environment.

The Guideline-as-Oracle Method: Guidelines Become an "Oracle"

The researchers took recommendations from the American Academy of Ophthalmology and compiled them into an operational table of 70 rows. Each row describes a condition by which a dialog can be assigned to the appropriate triage class. This table played the role of an "oracle": it automatically assigned labels to 3,000 training dialogs. Human annotation was needed only for final validation—it was used as a reference on a separate set of 201 clinical cases, but not during training.

Translating clinical texts into dialogs itself is a nontrivial engineering task. To make it reliable, the authors described eight corpus construction strategies. Among them are assigning a level according to the relevant guideline row, boundary pairs with "one fact differs," correcting errors only in metadata, and separately fixing labels. They characterized each strategy in terms of evidence: one works as an independent annotation mechanism, another has no effect, a third is confounded by other factors, and a fourth can only be evaluated as part of the entire pipeline.

This approach solves not only the cost problem but also the update problem: when clinical guidelines change, it is enough to adjust the rule table rather than re-annotate thousands of dialogs from scratch.

Experiments: Agreement Improved, Recall Jumped

The experiments were based on a model with 9 billion parameters. After fine-tuning on a corpus of 3,000 automatically annotated dialogs, an agent named GAO-Triage was produced. On the reference set of 201 cases, agreement with the expected decision increased from 61.7% to 74.1%. The change is statistically significant: McNemar's exact test gave p = 0.0046.

The growth in recall on unforeseen cases is especially notable. These are situations that do not fit the standard protocol—for example, when a patient describes symptoms requiring a non-template response. This is where agents typically make the most mistakes. For GAO-Triage, this metric increased from 9.5% to 69.0%, meaning the model became much less likely to miss potentially dangerous scenarios.

The improvements were not accidental: they persisted when retraining with a different seed and on a patient simulator. GAO-Triage was compared with seven general-purpose dialog systems. None of them dominated on both metrics simultaneously: some had higher agreement, others had higher recall, but only GAO-Triage combined both. Moreover, at the inference stage, the agent does not require the most powerful frontier model and can run locally on a 9B-parameter base.

The Secret to Success Lies in Label Assignment, Not in the Text

The authors ran a control experiment that explains why the method works. They took the same 3,000 dialogs but shuffled the correspondences between dialogs and the labels assigned by the rules. After this shuffling, the model completely degraded and turned into a constant routine predictor: it began producing the same standard response, ignoring the content of the conversation. This is an important result. It shows that the benefit comes not from the superficial form of utterances, but precisely from the connection between a specific dialog and the class derived from clinical rules.

Separately, the researchers tracked the label repair strategy—fixing labels at later stages of training. Its application coincided with the disappearance of safety degradation. Apparently, when annotation is carefully cleaned closer to the end of training, the model stops drifting toward unsafe responses that may arise from early errors in the data.

What This Means for Practice

The main takeaway of the study: in domains with mature clinical guidelines, a dialog agent can be trained with virtually no manual annotation. It is enough to translate the recommendations into executable rules once and then use them as a supervisor. A small expert sample is still needed—for final quality assessment—but its size is incomparable to fully annotating the training corpus.

The approach is not tied to ophthalmology. If similar protocols exist in cardiology, emergency care, or telemedicine, the scheme can be adapted there as well. The main technical challenge remains the same: clinical guidelines must be translated into unambiguous rules carefully and with domain expertise. But once that is done, training a safe medical agent becomes significantly faster and cheaper, and sensitive patient data remains under the developer's control.

Frequently asked questions

Related materials

All materials
Clinical rules instead of markup: a new method trains an ophthalmological triage agent on 3,000 dialogues