A research paper on arXiv proposes a tougher benchmark for "silicon sampling" — the practice of using large language models to stand in for human survey respondents.
Most existing evaluations check whether an LLM's outputs match the statistical distribution of a real survey population, which is a low bar. The researchers instead tested cross-survey transfer: give a model one set of a respondent's answers, then ask it to predict that same person's answers to completely different questions. Using Taiwan Election and Democratization Study data from 2024 and three open-weight models ranging from 27B to 120B parameters, they found zero-shot LLMs hit 52% accuracy on genuinely unseen items — within 6 percentage points of a supervised random forest trained on same-population data. That gap is smaller than most skeptics would have predicted.
The finding matters because silicon sampling is already being pitched as a cost-cutting substitute for expensive human panels. If the approach only works at the population level, it is a fancy way to regurgitate training data. Individual-level predictive accuracy is the harder and more honest test — and these results suggest the method clears a real, if modest, bar. Two caveats the field obsesses over — variance collapse and safety alignment effects — also turned out to be more complicated than assumed, with variance collapse showing up in supervised models too.
Predictability varied wildly by topic: partisan attitudes were forecast at 67% accuracy, while sovereignty questions landed at 23%, which should give anyone pause before deploying LLMs on politically sensitive or contested terrain.