[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-researchers-build-rulers-to-make-ai-judges-more-consistent":10,"sections":34},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":29,"feedback":33,"feedback_at":22,"cost_usd":33,"total_tokens":33},6367,"researchers-build-rulers-to-make-ai-judges-more-consistent","Researchers Build Rulers to Make AI Judges More Consistent","A new evaluation framework called Rulers aims to make AI judges score text more consistently and closer to how humans would rate it.","Researchers have built a system that tries to stop AI judges from grading the same piece of text differently depending on how the question is phrased.\n\nThe method, called Rulers, comes from a paper posted to arXiv. It locks a rubric's scoring criteria in place, requires the AI judge to cite specific evidence for each score rather than just spitting out a number, and then calibrates those scores against known human scoring boundaries. The team tested it across four rubric-governed benchmarks using multiple different underlying language models. Rulers matched human scores more closely than standard prompting in most cases, tracked the actual spread of human ratings instead of clustering around safe middle values, and held steady even when the same rubric was reworded in equivalent ways.\n\nLLM judges have quietly become the default way to grade everything from chatbot answers to coding benchmarks, mostly because paying humans to read thousands of outputs does not scale. But a judge that scores inconsistently, or cannot explain why it gave a 7 instead of an 8, is hard to trust or audit - which matters when that score decides which model ships. The paper's ablations suggest the fix is not a clever prompt trick; it is forcing the judge to show its work and calibrating the output, which is less flashy but more durable.\n\nGiven how much of the AI industry now runs on AI grading its own homework, a slightly more boring, more auditable judge is a bigger deal than it sounds.","[\"ai\",\"llm-evaluation\",\"ai-research\",\"benchmarks\"]","2026-09-11T04:00:00.000Z","2026-09-11T08:52:54.146Z","2026-09-11T08:53:06.070Z","published",null,[],"ai",[24,26,27,28],"llm-evaluation","ai-research","benchmarks",[30],{"name":31,"url":32},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2601.08654",0,{"sections":35},[36,39,43,47,52,57,62,65,70,74,79,84,89,94],{"name":37,"slug":24,"count":38,"latest_published_at":18},"AI",3543,{"name":40,"slug":41,"count":42,"latest_published_at":18},"Security","security",637,{"name":44,"slug":45,"count":46,"latest_published_at":18},"Policy","policy",338,{"name":48,"slug":49,"count":50,"latest_published_at":51},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":53,"slug":54,"count":55,"latest_published_at":56},"Hardware","hardware",153,"2026-09-09T15:12:32.000Z",{"name":58,"slug":59,"count":60,"latest_published_at":61},"Consumer Tech","consumer-tech",99,"2026-09-09T17:27:33.000Z",{"name":63,"slug":64,"count":60,"latest_published_at":18},"Science","science",{"name":66,"slug":67,"count":68,"latest_published_at":69},"Software","software",75,"2026-09-10T20:41:21.000Z",{"name":71,"slug":72,"count":73,"latest_published_at":18},"Dev Tools","dev-tools",70,{"name":75,"slug":76,"count":77,"latest_published_at":78},"Startups","startups",55,"2026-09-09T23:14:29.000Z",{"name":80,"slug":81,"count":82,"latest_published_at":83},"Gaming","gaming",43,"2026-09-10T12:18:06.000Z",{"name":85,"slug":86,"count":87,"latest_published_at":88},"General","general",41,"2026-09-08T01:57:23.000Z",{"name":90,"slug":91,"count":92,"latest_published_at":93},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":95,"slug":96,"count":97,"latest_published_at":98},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]