AI models pretending to be human survey respondents work better in bulk than up close.
Researchers tested large language models acting as "digital twins" — stand-ins for human respondents in psychological studies — against a construct validity framework designed to measure how well the models actually replicate human psychology. The short answer: well enough to be useful, but not well enough to be trusted without limits. Models scored high on aggregate accuracy and broad personality profile correlations, but item-level correlations were weaker. In word association tasks, the models produced humanlike network structure, yet diverged on specific word choices and local patterns. Crucially, when faced with decision-making tasks, the models leaned toward normative rationality rather than reproducing the cognitive shortcuts and biases humans routinely fall back on.
That gap matters because the whole pitch of digital twins is that they can substitute for human participants in research — cheaper, faster, scalable. If they sand down the very biases and variance researchers are trying to study, they are not substitutes; they are a cleaner dataset pretending to be a messy one. The finding that richer conditioning improved Big Five personality prediction is promising, but network invariance problems and persistent loading differences across languages suggest the models are approximating constructs, not embodying them.
This sits alongside a growing pile of research showing LLMs perform differently across languages and cultural contexts — the English-Chinese cross-language tests here reinforced that pattern. Until validity boundaries are mapped and published alongside any digital twin study, treat the output as a plausible sketch, not a human sample.