A research framework called MedEvoEval wants to know if AI doctor agents get better with experience — not just whether they answer correctly.
Most medical AI benchmarks hand a model a complete case and score its final diagnosis. MedEvoEval works differently. It runs agents through simulated outpatient episodes where evidence is only revealed when the agent takes a valid action — ordering an exam, requesting a consultation — mirroring how a real clinical encounter unfolds. The framework tracks a structured trace of every observation, action, and decision, then scores process costs alongside outcomes. It ships with 700 processed episodes, scoring scripts, and analysis code as a runnable artifact.
The distinction matters because an agent that reaches the right diagnosis after ordering every test in the catalog is not the same as one that reaches it efficiently. MedEvoEval also evaluates behavior across multiple episodes — whether agents retain earlier knowledge, transfer useful patterns to new cases, and respond to memory updates without forgetting what they already knew. That cross-episode lens is mostly absent from existing benchmarks.
The harder question MedEvoEval raises is whether any current model actually clears the bar — the paper shows the framework works, but the field still has no consensus on what "good" looks like for a longitudinal AI clinician.