[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-a-new-language-aims-to-standardize-ai-benchmarks":10,"sections":41},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":30,"tags":31,"sources":36,"feedback":40,"feedback_at":22,"cost_usd":40,"total_tokens":40},8181,"a-new-language-aims-to-standardize-ai-benchmarks","A New Language Aims to Standardize AI Benchmarks","A new spec called Benchy tries to make AI benchmarks a shared, checkable format instead of ad hoc scripts and scoreboards.","A team of researchers has proposed Benchy, a formal language and execution engine meant to make AI benchmarks precise enough for machines to check, not just humans to eyeball.\n\nBenchy defines a benchmark as three fixed pieces: a program, a scoring function, and a dataset, kept separate from whichever AI system takes the test. Benchmarks are written in a canonical YAML format, tagged against a shared task\u002Fdomain\u002Flanguage ontology, then compiled into a JSON intermediate representation that the engine actually runs. That compilation step only changes format, the paper stresses, not meaning: it won't quietly patch a broken benchmark definition or fill in defaults you didn't ask for. Every AI system plugs into the engine through one fixed contract, a named-field input in, a named-field output out, so the messy work of hooking up a model never leaks into how the benchmark is scored.\n\nThe pitch here is less about any single benchmark and more about the format wars underneath all of them. Right now, benchmark suites are typically bespoke code, which makes results hard to audit and easy to game with hidden preprocessing. A shared, machine-checkable spec would let anyone inspect exactly what a benchmark measures and how a score gets computed, rather than trusting a leaderboard's word for it.\n\nWhether Benchy becomes that shared spec or just another one competing for attention is the open question. The paper itself only commits to the engineering contract for a first engine implementation, which is a modest way of saying: this is a proposal, not yet a standard.","[\"ai-benchmarks\",\"evaluation\",\"research\",\"dev-tools\"]","2026-09-28T04:00:00.000Z","2026-09-28T13:02:45.078Z","2026-09-28T13:02:51.812Z","published",null,[24],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"Drop or justify the 'open-source' tag — neither the article nor the source paper says Benchy's code, engine, or spec will be released as open source, so the tag implies an unsupported claim.","resolved","ai",[32,33,34,35],"ai-benchmarks","evaluation","research","dev-tools",[37],{"name":38,"url":39},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.30550",0,{"sections":42},[43,46,50,55,60,65,69,74,79,83,88,93,97,102],{"name":44,"slug":30,"count":45,"latest_published_at":18},"AI",4798,{"name":47,"slug":48,"count":49,"latest_published_at":18},"Security","security",762,{"name":51,"slug":52,"count":53,"latest_published_at":54},"Policy","policy",399,"2026-09-27T18:39:02.000Z",{"name":56,"slug":57,"count":58,"latest_published_at":59},"Deals","deals",261,"2026-09-27T15:30:35.000Z",{"name":61,"slug":62,"count":63,"latest_published_at":64},"Hardware","hardware",188,"2026-09-27T20:46:36.000Z",{"name":66,"slug":67,"count":68,"latest_published_at":18},"Science","science",151,{"name":70,"slug":71,"count":72,"latest_published_at":73},"Consumer Tech","consumer-tech",135,"2026-09-26T14:30:00.000Z",{"name":75,"slug":76,"count":77,"latest_published_at":78},"Software","software",91,"2026-09-25T20:55:00.000Z",{"name":80,"slug":35,"count":81,"latest_published_at":82},"Dev Tools",84,"2026-09-26T04:20:58.000Z",{"name":84,"slug":85,"count":86,"latest_published_at":87},"Startups","startups",76,"2026-09-25T18:33:59.000Z",{"name":89,"slug":90,"count":91,"latest_published_at":92},"Gaming","gaming",48,"2026-09-25T18:35:21.000Z",{"name":94,"slug":95,"count":91,"latest_published_at":96},"General","general","2026-09-26T17:02:42.000Z",{"name":98,"slug":99,"count":100,"latest_published_at":101},"Reviews","reviews",30,"2026-09-24T20:07:31.000Z",{"name":103,"slug":104,"count":105,"latest_published_at":106},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]