The same AI model can give you different safety answers depending on whether you type into the chat window or hit the API directly.
Researchers tested ChatGPT across two access points, the consumer chat UI and OpenAI's API, each with web search turned on or off. That is four conditions, run three times per prompt, across a stratified set of 401 prompts pulled from two safety benchmarks, BBQ and SafetyBench, for a total of 4,812 responses. With search disabled, the chat UI was less accurate than the API on both benchmarks. Turning on web search knocked up to 8 percentage points off accuracy and, on one benchmark, flipped which modality came out ahead.
Benchmarks are the main evidence companies and regulators point to when calling a model safe or deployment ready, and those tests usually run once, through one access point, with no search involved. This paper shows that setup measures a narrow slice of how people actually use these tools. The same prompt, run three times, produced inconsistent answers in up to 21% of cases, and the two modalities cited different sources and declined to answer in different patterns.
A benchmark score is a snapshot of one configuration, not a guarantee about the product you actually open every morning.