How pointwise LLM prompting uncovers hidden medical data that standard de-identification systems miss

Why Standard De-identifiers Miss "Local" Data

Automated de-identification systems for medical records usually handle typical types of protected information well: names, dates of birth, insurance numbers. But in a real hospital, there is data that is sensitive only within that organization. This includes abbreviated clinic names, building names, and internal department codes. To a human, such strings unambiguously indicate the place of treatment, but a universal algorithm cannot know all local designations.

Traditional solutions rely either on dictionaries or on trained models with fixed categories. Both miss what is not in their reference sets. Researchers call this "institutionally localized" PHI, and this is exactly where serious leakage risks arise.

How LLMs Bridge the Gap

To test whether large language models with in-context learning can solve this problem, the experiment's authors took 100 annotated pediatric oncology notes — over five thousand PHI fragments in total — and then compared eight LLMs against two specialized systems (Stanford TiDE and OpenMed PII) and two pattern baselines.

The result favored the LLMs: the best model achieved F1 = 0.918 ± 0.001, while the best specialized system, TiDE, only reached 0.779. The advantage was especially noticeable on context-dependent categories — precisely those local designations that usually fall outside the algorithms' scope.

Three Levels of Prompts: From HIPAA to Fine-Tuning

The key technique is not just a query, but a deliberate prompt configuration tailored to a specific institution. The experiment used three variants:

• Basic prompt — general instructions per the HIPAA standard.
• Prompt with local categories — adds a list of institutional PHI types that the standard does not cover.
• Prompt with protection against over-redaction — an additional instruction not to cut out excess clinical content.

Listing the missed categories recovered 79% of the 61 previously unfound fragments (48 items). And the instruction against over-redaction restored the precision that had suffered from the model's "over-caution."

Agents and Ensembles Offered No Gain

One might assume that running multiple models or a multi-agent scheme would yield even better results. The authors tested 14 such configurations and compared them against the best single prompt. It turned out that no agent architecture outperformed a simple single-pass calibrated prompt. Agents held F1 in the 0.906–0.907 range — essentially the same level, but without any additional gain.

LLMs as Annotation Auditors

A separate value of the LLM approach is that it helps verify the quality of the reference set itself. The models flagged 414 places where the original annotation was likely incomplete. Manual review confirmed 227 missed PHI spans. After fixing the annotation and re-running the prompting, recall rose to 0.981 with F1 = 0.907 ± 0.002. This means an LLM can serve not only as a de-identification tool, but also as an auditor for building more accurate training sets.

Conclusions

A well-tuned prompt allows an LLM to close the institutional PHI gap and find the optimal precision-recall balance in a single call to the model. This approach is more expensive than traditional dictionary-based methods, but those costs are partially offset by the ability to verify and improve the reference standard itself.

The authors consider LLMs a legitimate, adaptable alternative to specialized de-identifiers. The main implementation strategy is developing institution-specific prompts that account for each clinic's local realities.