A new research framework shows that when AI chatbots seem to agree, they are often just echoing the same safe default answer.
Researchers built CHOIR, short for Collective Hierarchically-Ordered Inquiry Responses, a method adapted from free-list elicitation, a technique borrowed from cognitive anthropology, to test whether agreement among large language models reflects genuine consensus or just a narrow answer space. The tool repeatedly asks models to generate ranked lists of answers, clusters the results into concepts, and tracks how often each concept appears across different models, prompt phrasings, and assigned personas. Tested on a 100-prompt open-ended benchmark called Infinity-Chat and a separate 27-question diagnostic set, the framework found that 93 of 100 prompts produced agreement well above chance. Look closer, though, and that agreement splits into two patterns: narrow prompts converge hard, while broad prompts hide varied, recoverable alternatives beneath the surface.
The finding undercuts a common assumption - that running one question past several chatbots gives you independent perspectives. In practice, a model's base identity, not its assigned persona, is the strongest predictor of what it will say. Swapping personas shifts which ideas surface, not the underlying menu of options available to the model.
If you want a genuinely different take from AI, changing its personality prompt won't do it - you need a different base model entirely.