A new study finds that the people designing AI 'moral voting' systems shape the outcome long before anyone casts a vote.
Researchers ran a two-phase study with 809 participants across three scenarios: allocating kidneys via AI, AI agents standing in for absent workers, and AI-generated depictions of the deceased. They tested three decisions developers make before collecting any votes: which features get put to a vote, who gets sampled as a voter, and how the question gets worded. The morally relevant features shifted depending on context, meaning a feature list built for one use case does not automatically transfer to another. About a third of features showed preference differences tied to political ideology, with some flipping direction entirely. Question wording alone could shift ideological gaps by up to a full point on the study's scale.
This matters because vote-based AI alignment gets pitched as a neutral, democratic alternative to a company simply deciding what's ethical on its own. The study suggests that framing doesn't hold up. Every step before the vote, from who's invited to answer to how the question is phrased, already tilts the result.
Aggregating opinions was never going to erase bias. It just relocates the bias to the survey design.