[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-aligning-ai-to-human-preferences-makes-it-less-human-like":10,"sections":35},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":30,"feedback":34,"feedback_at":22,"cost_usd":34,"total_tokens":34},7291,"aligning-ai-to-human-preferences-makes-it-less-human-like","Aligning AI to Human Preferences Makes It Less Human-Like","New research finds preference-aligned AI often sounds less human, not more, even when trained entirely on human data.","Teaching an AI to give the answers people prefer is not the same as teaching it to answer the way people actually would.\n\nA new arXiv paper draws that line explicitly. The researchers distinguish \"alignment with human preferences\" from \"alignment with human behavior\" and find the two pull in different directions. Even when both the preference judgments and the training responses come entirely from humans, optimizing for what people say they like pushes a model away from how people actually talk. They call this the Turing-test gap. Preference alignment only preserves a human-like response distribution under a narrow mathematical condition, and the paper finds no evidence real human preferences satisfy it. The more heavily a model is weighted toward preferred answers, regardless of direction, the further its output drifts from genuine human responses, and the same gap shows up under standard DPO, the preference-tuning method built into most major model training pipelines.\n\nThat is an awkward result for an industry that markets \"aligned\" models as human-like by design. Preference alignment (RLHF, DPO, and their variants) is the step meant to turn a raw language model into something that sounds like a helpful person. This paper argues that step, done at scale, actively erodes the resemblance it is supposed to protect. It reframes human-likeness as something that has to be measured and optimized for on its own terms, not a side effect labs get for free.\n\nWorth remembering next time a chatbot's tone gets praised as \"so human\": it may be optimized to please you, not to sound like anyone real.","[\"ai-alignment\",\"llm\",\"rlhf\",\"dpo\"]","2026-09-23T04:00:00.000Z","2026-09-23T05:21:43.045Z","2026-09-23T05:21:47.378Z","published",null,[],"ai",[26,27,28,29],"ai-alignment","llm","rlhf","dpo",[31],{"name":32,"url":33},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.23640",0,{"sections":36},[37,40,44,49,54,59,63,68,73,78,83,88,93,98],{"name":38,"slug":24,"count":39,"latest_published_at":18},"AI",4248,{"name":41,"slug":42,"count":43,"latest_published_at":18},"Security","security",706,{"name":45,"slug":46,"count":47,"latest_published_at":48},"Policy","policy",369,"2026-09-23T02:13:52.000Z",{"name":50,"slug":51,"count":52,"latest_published_at":53},"Deals","deals",202,"2026-09-22T23:00:04.000Z",{"name":55,"slug":56,"count":57,"latest_published_at":58},"Hardware","hardware",168,"2026-09-22T23:56:03.000Z",{"name":60,"slug":61,"count":62,"latest_published_at":18},"Science","science",133,{"name":64,"slug":65,"count":66,"latest_published_at":67},"Consumer Tech","consumer-tech",110,"2026-09-22T20:00:00.000Z",{"name":69,"slug":70,"count":71,"latest_published_at":72},"Software","software",80,"2026-09-22T23:32:52.000Z",{"name":74,"slug":75,"count":76,"latest_published_at":77},"Dev Tools","dev-tools",79,"2026-09-22T22:21:13.000Z",{"name":79,"slug":80,"count":81,"latest_published_at":82},"Startups","startups",65,"2026-09-22T22:06:48.000Z",{"name":84,"slug":85,"count":86,"latest_published_at":87},"Gaming","gaming",45,"2026-09-22T15:35:06.000Z",{"name":89,"slug":90,"count":91,"latest_published_at":92},"General","general",43,"2026-09-21T23:48:56.000Z",{"name":94,"slug":95,"count":96,"latest_published_at":97},"Reviews","reviews",27,"2026-09-22T13:00:00.000Z",{"name":99,"slug":100,"count":101,"latest_published_at":102},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]