Standard psychometric tests for AI personas throw away the most informative data.
Researchers ran GPT-4o through the IPIP-50 personality questionnaire while simulating American and Chinese-American personas, then scrambled the question order to see what held up. They measured two separate things: the usual Big Five aggregate scores, and a geometric structure built from correlations between individual responses. Shuffling question order knocked aggregate scores down 21%. Geometric structure fared worse - a 42% drop under misaligned framing - but when question framing was consistent, it recovered to 84% accuracy, beating the aggregates at 76%. The conclusion: these are two distinct signals, not one.
The finding matters because most AI persona research stops at aggregate scores, which are the psychological equivalent of averaging a distribution and calling it done. The geometric approach captures coordination patterns between responses - how answers relate to each other, not just where they land on a scale. That relational structure turns out to encode information the averages simply cannot see.
The practical implication cuts both ways. Designers building AI personas for products, and researchers evaluating whether those personas are stable or culturally consistent, have been working with an incomplete instrument. The less flattering read: if LLM persona geometry is this sensitive to question ordering, the "personality" being measured may say as much about the test as the model.