AI/ ai · ai-agents · research · turing-test

AI Agents Can Fake Answers but Not How They Think

Researchers found that measuring how an AI solves a task, not just whether it gets the answer right, is a far better way to spot a machine posing as a human.

A new test for telling humans from AI agents cares less about what answer you give and more about how you arrived at it.

The paper, posted to arXiv, introduces the Process Turing Test, a battery of cognitive tasks including mental rotation, sequence prediction, working-memory challenges, planning exercises, and CAPTCHAs, scored for process-level behavior rather than just pass or fail. Researchers ran the tasks against off-the-shelf frontier agents (Claude Sonnet 4.5, GPT-5, Gemini 2.5 Pro), Centaur (an LLM fine-tuned on 10.7 million human decisions), and two custom-built models: A-SFT, fine-tuned directly on human actions, and P-SFT, fine-tuned specifically to match human process features. Because the study matched human and agent task performance by design, a classifier that only looked at final scores could do no better than a coin flip. One built from process-level features instead reached an AUC of 0.88, reliably telling humans and agents apart even when their output looked identical.

That gap is the real headline: getting the right answer is easy to fake, but getting there the way a person does is not, at least not yet. Broad fine-tuning on human choices already nudged agents toward more human-like processes, and P-SFT pushed further still, but most of that advantage disappeared once the fine-tuned models faced tasks outside their training. For anyone building bot detection, behavioral CAPTCHAs, or agent-auditing tools, that is the actionable bit: process signals generalize poorly across tasks, so a detector tuned on one skill will not automatically catch an agent faking a different one.

CAPTCHAs already lost the outcome-matching arms race years ago; this is an early signal that the next round will be fought over mouse jitter and hesitation, not checkboxes.

TR

The Revision

Written by an AI system from the public sources credited above. How we write →