A new benchmark finds that the best available method for guessing a missing medical code gets it right on the first try only 15 percent of the time.
The paper, arXiv:2609.17855 ('SNOMED CT Concept Recommendation from Masked Clinical Context'), builds its test from the SNOMED CT Entity Linking Challenge dataset, itself derived from MIMIC-IV-Note discharge summaries. Researchers masked a clinical term from its surrounding context in 75,491 note-concept pairs across 272 discharge summaries, then asked several systems to guess the missing SNOMED CT concept from a candidate list built during training. A simple TF-IDF model beat every fancier alternative, including a retrieval-augmented hybrid, hitting Recall@1 of 14.81%, Recall@10 of 33.43%, and an MRR of 0.2114. Retrieval augmentation actually made results worse, dropping Recall@10 to 31.99%.
That gap matters because SNOMED CT is the backbone of clinical interoperability - it's how hospitals, insurers, and researchers are supposed to talk about the same diagnosis without agreeing on the words. The benchmark shows performance collapses for anything but common terms: Recall@10 falls to 7.74% for concepts seen only once or twice in training, versus 43.90% for well-represented ones, and nearly 1 in 10 test concepts never showed up in training at all.
In other words, coding tools are only as good as the paperwork that trained them, and no amount of retrieval-augmented machinery fixed that here - a useful reality check before anyone pitches an 'AI coding assistant' as ready for rare diagnoses.