Anthropic has published guidance on building reliable AI safety tests, and its headline example is a gap: Claude Fable 5.1 behaved appropriately in 60% of self-harm test conversations through the API, against 94% in the Claude app after a fix.
The API is the route developers use to build their own products on top of the model. The app is Anthropic's own consumer product. The two numbers are 34 percentage points apart, but they are not a clean comparison. The 94% comes after a fix, so the difference mixes the access route with whatever was repaired. Anthropic's guidance, as described, does not say what the fix was.
The company also warned that many headline safety scores hide wide uncertainty. In plain terms, a single percentage can sit on top of a broad range of plausible results, depending on how the test was built and run.
That matters because safety scores usually travel as single numbers: in launch announcements, in comparisons between labs, in the tables readers skim. A figure like 60% or 94% looks precise, and it invites ranking. The guidance is a reminder that the useful questions come before the number: how many conversations were tested, how they were written, how appropriate behavior was judged, and how much the result moves when any of that changes.
The closest everyday comparison is polling. A poll that reports 52% with no margin of error is hard to interpret, and a safety score with no stated uncertainty has the same problem. The material available here does not say how many conversations Anthropic tested or how wide its uncertainty was, which is the detail that would make either figure easier to read.