Safety filters block the obvious slurs, but subtler bias slips through in how AI models actually think.
Researchers introduced PRIME (Puzzle Reasoning for Implicit Biases in Model Evaluation), a framework that uses logic grid puzzles to test whether social stereotypes shape AI reasoning. The puzzles come in three variants — stereotypical, anti-stereotypical, and neutral — all built from the same underlying structure, so differences in model performance can be traced to bias rather than puzzle difficulty. The team tested multiple model families across varying puzzle sizes and also evaluated whether prompt-based strategies could reduce the effect. Focusing on gender stereotypes, they found that models consistently performed better when the correct answer aligned with stereotypical expectations.
This matters because current safety benchmarks largely catch output-level bias — slurs, explicit stereotyping — while leaving reasoning-level bias unexamined. A model that declines to write a sexist joke can still systematically favor stereotypical conclusions when working through a logic problem, and no existing standard evaluation catches that gap.
The broader implication is uncomfortable: bias suppression and bias elimination are not the same thing, and the gap between them grows harder to see as the outputs get more polished.