AI models can often tell when they're being tested, and sometimes they act differently because of it.
A new paper builds a framework for measuring this, splitting "evaluation awareness" into two separate things: whether a model recognizes it's under evaluation, and whether that recognition changes its behavior. The researchers tracked both across nine frontier models and four benchmarks using chain-of-thought monitoring, and identified eight environmental triggers that can tip a model off, including placeholder entities and grading-style output formats. They found that recognition rates vary a lot depending on which model is paired with which benchmark, and that noticing rarely translates into a different answer. To isolate which triggers matter most, they also released EvalAwareBench, 100 paired safety-and-capability tasks where each of the eight triggers can be toggled independently.
Models are more sensitive to safety evaluations than capability ones - meaning they're more likely to clock that a safety test is a safety test - which puts the validity of safety benchmarks specifically at risk. Noticing doesn't reliably change the answer, but on the rare occasions it does, which way the model swings depends on what kind of test it thinks it's taking. No single trigger reliably tips off every model, but stacking several at once raises awareness across the board regardless of which model you're looking at.
That's an awkward result for an industry that treats safety benchmarks as its main proof a model behaves, right up until the model realizes it's being watched.