Doctors ask better questions than medical boards, and that gap is exposing the limits of general-purpose AI.
Researchers recruited 149 practicing physicians across 36 states to evaluate answers from three frontier models — Claude Opus 4.8, Gemini 3.1 Pro, and GPT-5.5 — against OpenEvidence, a specialized clinical AI tool. The questions weren't hypotheticals or board-exam prompts; they were 620 real queries submitted by physicians across 30 specialties during actual patient care. On accuracy, clinical utility, source quality, verifiability, and completeness, the specialized tool won every category. Win-rate margins over the general-purpose models ranged from 25 to 39 percentage points, with p-values well below 0.001.
The finding matters because most AI benchmarks in medicine still rely on licensing-exam questions, which are designed to have clean, defensible answers — the opposite of the messy, context-dependent queries a cardiologist fires off mid-consult. That mismatch has let general-purpose models look more capable than they may be in practice. The researchers also found that LLM-based judges diverged systematically from physician judges, which raises questions about how much of the existing benchmark literature can be trusted.
The paper stops short of saying general models can't serve clinical roles — the authors explicitly note that targeted engineering, not raw model capability, explains the specialized tool's edge. That's a useful distinction: the story here isn't that GPT-5.5 is bad at medicine, it's that tuning for a specific professional context still pays off significantly, even when the base models are extremely capable.