Three open-source LLMs were put through 121 game-theory experiments — and they behaved very differently from each other.
Researchers tested Llama, Mistral, and Qwen across four classical game types to see how closely each model mirrors human decision-making. Llama reproduced human cooperation patterns with high fidelity and shared what the study calls an "envious" decision profile with human participants. Qwen, by contrast, aligned closely with Nash equilibrium predictions — the coldly rational outcome game theorists calculate on paper. Mistral exhibited its own distinct behavioral profile, separate from both human-like cooperation and pure Nash logic. An attention-based analysis added a mechanistic angle: Llama processes payoff information in a structured, layer-dependent way that Qwen and Mistral both lack, which the researchers point to as a likely explanation for Llama's closer alignment with human behavior.
The practical stakes are real. LLMs are already being deployed as decision-making agents in high-stakes settings and as stand-ins for human participants in social-science simulations — a use that quietly assumes the models actually behave like people. This study suggests that assumption holds for some models and fails for others, which matters a lot depending on which one you pick.
Notably, the human-level replication happened without persona-based prompting, which simplifies the simulation pipeline considerably — though it also raises the question of why Llama converged on human behavior in the first place, a question the attention analysis starts to answer but does not close.