AI agents built on large language models can fake it at the population level — but not at the personal one.
Researchers tested nine open-weight LLMs against a large-scale networked Prisoner's Dilemma experiment previously run with real human participants. They held everything constant — the interaction protocol, payoff structure, and network topologies — so any gap would land squarely on the models. One model did reproduce some macro-level cooperation dynamics, including an early drop in cooperation followed by stabilization. That sounds like a win until you look closer.
The models systematically underestimated how differently individual humans behave from one another, and they produced conditional cooperation patterns that did not match human decision rules. Adding a sprinkle of random agents improved some micro-level metrics but could not fix the underlying mismatch. The researchers call it a "macro-micro dissociation": collective outcomes can look convincingly human even when the behavioral machinery underneath is not.
This matters because AI agents are increasingly being used as stand-ins for people in social simulations — to model policy impacts, study network dynamics, or prototype economic experiments. If the aggregate output validates but the individual behavior diverges, researchers risk drawing confident conclusions from a model that is right for the wrong reasons. Outcome-level agreement, the paper argues, is not enough; validation needs to check aggregate dynamics, individual heterogeneity, and context-dependent decision rules together. Social science has spent decades worrying about whether surveys capture real behavior. Now it has to ask the same question about AI.