A linguistics-grounded benchmark wants to make AI safety evaluations harder to fake.
Researchers have published a framework called adversarial pragmatics, designed to stress-test how we measure whether a language model actually follows instructions, refuses harmful requests, or resists embedded commands. The paper introduces a taxonomy of failure types — instruction conflict, scope ambiguity, indirect speech acts, prompt injection, and multi-turn agent transcripts, among others — that existing benchmarks tend to collapse into a single pass/fail label. The researchers built an 18-item seed benchmark and a 54-row pilot, each with validator-enforced metadata, plus an expert-evaluation protocol that separately scores task success, policy compliance, safety risk, refusal outcome, and evaluator confidence.
The stakes are higher than they look. Most public safety benchmarks grade models with automated LLM judges, which have their own blind spots around language ambiguity. If the judge cannot distinguish between a model that refused because it understood the risk and one that refused because it misread the instruction, the score is noise — and safety documentation built on that noise is a false floor. This framework gives researchers a way to audit both the model and the evaluator at once.
The deeper problem the paper surfaces is that safety evaluation is itself a language task, and natural language is slippery. A model might ace a benchmark by pattern-matching on surface cues rather than genuinely tracking policy intent. Whether this seed benchmark scales into something the broader research community adopts — or stays a methodological paper with a tidy taxonomy — depends entirely on who picks it up next.