[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-a-cleaner-way-to-train-llms-on-messy-human-rankings":10,"sections":34},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":29,"feedback":33,"feedback_at":22,"cost_usd":33,"total_tokens":33},3400,"a-cleaner-way-to-train-llms-on-messy-human-rankings","A Cleaner Way to Train LLMs on Messy Human Rankings","Researchers propose a ranking-robust alignment method that cuts through annotator noise without sacrificing performance on clean data.","A new alignment technique targets one of the quieter headaches in language model training: what happens when the humans doing the ranking can't quite agree.\n\nResearchers have proposed a distributionally robust variant of the Plackett-Luce ranking objective — the math behind listwise preference optimization — that accounts for uncertainty in ranking labels. The problem it addresses is specific: when you show annotators a list of model outputs and ask them to rank them, near-ties, inconsistent raters, and noisy reward models can make those rankings unreliable. The proposed method treats that uncertainty explicitly, adding a worst-case correction term on top of the standard loss. Crucially, solving for that worst-case ranking turns out to be equivalent to sorting outputs by their current scores in ascending order, which drops the computational cost from factorial to O(K log K).\n\nMost alignment robustness work has focused on the pairwise setting — comparing two outputs at a time — or on cleaning up datasets and prompts before training. This approach instead bakes robustness into the ranking step itself, which is where listwise methods like those used in RLHF pipelines are most exposed to label noise. The method comes with convergence guarantees in both offline and online settings, and experiments show it holds performance on clean labels while improving on noisy ones.\n\nLLM alignment is still largely a craft discipline, with labs tuning reward models and data pipelines by intuition as much as theory — a tractable, guaranteed-robust ranking objective is the kind of unglamorous infrastructure work that tends to matter more than the demos suggest.","[\"ai\",\"machine-learning\",\"alignment\",\"llm\"]","2026-07-03T04:00:00.000Z","2026-07-03T05:03:38.527Z","2026-07-03T05:03:41.494Z","published",null,[],"ai",[24,26,27,28],"machine-learning","alignment","llm",[30],{"name":31,"url":32},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.01715",0,{"sections":35},[36,40,45,50,55,60,65,70,75,80,85,89,94,99],{"name":37,"slug":24,"count":38,"latest_published_at":39},"AI",2590,"2026-07-16T04:00:00.000Z",{"name":41,"slug":42,"count":43,"latest_published_at":44},"Security","security",294,"2026-07-15T19:59:48.000Z",{"name":46,"slug":47,"count":48,"latest_published_at":49},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":51,"slug":52,"count":53,"latest_published_at":54},"Policy","policy",158,"2026-07-16T00:02:48.000Z",{"name":56,"slug":57,"count":58,"latest_published_at":59},"Hardware","hardware",122,"2026-07-14T19:46:26.000Z",{"name":61,"slug":62,"count":63,"latest_published_at":64},"Consumer Tech","consumer-tech",93,"2026-07-13T13:20:48.000Z",{"name":66,"slug":67,"count":68,"latest_published_at":69},"Software","software",70,"2026-07-13T19:52:25.000Z",{"name":71,"slug":72,"count":73,"latest_published_at":74},"Science","science",66,"2026-07-10T10:29:37.000Z",{"name":76,"slug":77,"count":78,"latest_published_at":79},"Dev Tools","dev-tools",59,"2026-07-07T04:00:00.000Z",{"name":81,"slug":82,"count":83,"latest_published_at":84},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":86,"slug":87,"count":83,"latest_published_at":88},"Startups","startups","2026-06-29T20:55:50.000Z",{"name":90,"slug":91,"count":92,"latest_published_at":93},"General","general",29,"2026-07-10T22:28:58.000Z",{"name":95,"slug":96,"count":97,"latest_published_at":98},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":100,"slug":101,"count":102,"latest_published_at":103},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]