Researchers have released MedRepBench, a public benchmark designed to measure how accurately AI models extract and explain structured data from real-world medical report images.
The dataset covers 1,925 de-identified Chinese medical report images drawn from multiple hospital departments, patient demographics, and document formats. The benchmark focuses narrowly on interpretation — pulling out fields like item, value, unit, reference range, and abnormal flag — rather than diagnostic reasoning or treatment suggestions. Evaluation runs on two tracks: an objective protocol that counts field-level recall, and a subjective protocol where an LLM judge scores factuality and clarity. The researchers also tested an OCR-plus-LLM pipeline as a ceiling estimate for when character recognition is clean.
The gap matters because structured extraction from scanned clinical documents is a prerequisite for any real patient-facing tool or hospital system integration. Most existing benchmarks test general document understanding, not the specific schema a discharge summary or lab report demands — so strong scores elsewhere don't translate.
As a proof of concept, the team fine-tuned a mid-sized vision-language model using a reinforcement learning method called GRPO, lifting field-level recall by up to 6%. That's a modest gain, and the OCR pipeline analysis shows layout errors and latency add meaningful friction — a quiet reminder that "AI reads your lab results" is further from clinical reality than the pitch decks suggest.