Medical AI looks shakier when tested against the data formats hospitals actually use.
Researchers built a pipeline that converts unstructured clinical case text into HL7 FHIR R4 bundles — the structured, interoperable format most modern electronic health record systems run on. They applied it to an existing dataset of clinician-authored diagnostic cases, producing MedCase-Structured, a synthetic benchmark with valid FHIR output for 82.5% of cases. The pipeline uses staged LLM generation plus terminology-grounded validation to catch and repair hallucinated medical codes before they corrupt the output.
The uncomfortable finding: LLMs score consistently lower on diagnostic accuracy when given structured FHIR inputs than when given the same information as plain text. That gap matters because nearly every real clinical deployment hands AI structured records, not narrative prose. Benchmarks built on unstructured text are quietly testing a scenario that rarely exists in practice.
Most published clinical AI results lean on static, text-based datasets — cleaner, cheaper, and more flattering than what a real EHR spits out. MedCase-Structured is an early attempt to close that gap, but 17.5% of cases still failing valid FHIR generation is a reminder that even the benchmark itself is a work in progress.