[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-new-medical-ai-benchmark-finds-safety-gaps-across-top-models":10,"sections":46},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":34,"persona_id":22,"persona_name":22,"section":35,"tags":36,"sources":41,"feedback":45,"feedback_at":22,"cost_usd":45,"total_tokens":45},2427,"new-medical-ai-benchmark-finds-safety-gaps-across-top-models","New Medical AI Benchmark Finds Safety Gaps Across Top Models","IMCBench tests eight AI models on multi-turn clinical image conversations and finds that accuracy alone does not predict safe patient guidance.","A new benchmark finds that frontier AI models can discuss medicine from a scan, but keeping patients safe is another matter.\n\nResearchers introduced IMCBench, a benchmark that pairs real, publicly available clinical images with synthetic patient profiles to simulate multi-turn patient-clinician exchanges. Eight models across four families (Claude, GPT, Nova, and Llama) were scored on safety, accuracy, and calibrated uncertainty on a 1-5 scale, using an LLM-as-jury system calibrated against expert clinician annotations. The paper identifies the top performer as a model its authors designate Claude Opus 4.6 (3.61 overall), followed by models they label Claude Sonnet 4.6 (3.30) and GPT-5.2 (3.29); those are the researchers' own designations and have not been independently verified against official product lineups. No model led on every dimension.\n\nThe headline finding is not the leaderboard order; it is the safety drop. Scores fell by 0.27 points for both malignant and rare conditions, and removing either visual input or electronic health record context made models measurably less safe (average drops of 0.18 and 0.23 respectively). That gap between accuracy and safe guidance is precisely what earlier single-turn, text-only benchmarks could not surface.\n\nMost medical AI evaluations test whether a model can answer a question correctly. IMCBench tests whether it would give advice a clinician would stand behind. Current frontier models have not cleared that bar.","[\"medical ai\",\"benchmarks\",\"language models\",\"safety\"]","2026-06-30T04:00:00.000Z","2026-06-30T05:07:15.178Z","2026-06-30T05:07:22.348Z","published",null,[24,30],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"The article cites 'Claude Opus 4.6', 'Claude Sonnet 4.6', and 'GPT-5.2' as specific model names — these product identifiers cannot be confirmed to exist in their respective companies' actual lineups (Claude Opus 4.6 and GPT-5.2 are not known released models), and while the source material uses the same names, publishing unverifiable proper product names that appear fabricated or speculative fails the product-name verification check; the writer should flag this discrepancy to editors before repub","resolved",{"id":31,"reviewer":26,"round":32,"reason":33,"status":29},"editor-r2",2,"The inline editorial flag ('Editorial flag: the paper names specific model identifiers…') is a visible editorial artifact that must be removed before publication; beyond that, the open concern [editor-r1] remains unresolved — the draft still references 'Claude Opus 4.6,' 'Claude Sonnet 4.6,' and 'GPT-5.2' as named model identifiers that cannot be confirmed to exist in their respective companies' actual lineups, and the writer must either obtain verification or explicitly attribute these names so","https:\u002F\u002Fcdn.xyz.onl\u002Farticle-images\u002Fnew-medical-ai-benchmark-finds-safety-gaps-across-top-models.webp","ai",[37,38,39,40],"medical ai","benchmarks","language models","safety",[42],{"name":43,"url":44},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.28556",0,{"sections":47},[48,52,57,62,67,72,77,82,87,92,97,101,106,111],{"name":49,"slug":35,"count":50,"latest_published_at":51},"AI",2590,"2026-07-16T04:00:00.000Z",{"name":53,"slug":54,"count":55,"latest_published_at":56},"Security","security",294,"2026-07-15T19:59:48.000Z",{"name":58,"slug":59,"count":60,"latest_published_at":61},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":63,"slug":64,"count":65,"latest_published_at":66},"Policy","policy",158,"2026-07-16T00:02:48.000Z",{"name":68,"slug":69,"count":70,"latest_published_at":71},"Hardware","hardware",122,"2026-07-14T19:46:26.000Z",{"name":73,"slug":74,"count":75,"latest_published_at":76},"Consumer Tech","consumer-tech",93,"2026-07-13T13:20:48.000Z",{"name":78,"slug":79,"count":80,"latest_published_at":81},"Software","software",70,"2026-07-13T19:52:25.000Z",{"name":83,"slug":84,"count":85,"latest_published_at":86},"Science","science",66,"2026-07-10T10:29:37.000Z",{"name":88,"slug":89,"count":90,"latest_published_at":91},"Dev Tools","dev-tools",59,"2026-07-07T04:00:00.000Z",{"name":93,"slug":94,"count":95,"latest_published_at":96},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":98,"slug":99,"count":95,"latest_published_at":100},"Startups","startups","2026-06-29T20:55:50.000Z",{"name":102,"slug":103,"count":104,"latest_published_at":105},"General","general",29,"2026-07-10T22:28:58.000Z",{"name":107,"slug":108,"count":109,"latest_published_at":110},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":112,"slug":113,"count":114,"latest_published_at":115},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]