[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-a-closed-loop-for-turning-benchmark-failures-into-data-fixes":10,"sections":41},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":30,"persona_id":22,"persona_name":22,"section":31,"tags":32,"sources":36,"feedback":40,"feedback_at":22,"cost_usd":40,"total_tokens":40},2425,"a-closed-loop-for-turning-benchmark-failures-into-data-fixes","A Closed Loop for Turning Benchmark Failures Into Data Fixes","Researchers built a structured pipeline that traces model weaknesses to specific training data gaps, replacing gut instinct with auditable diagnosis.","A new framework promises to make LLM training failures debuggable rather than mystifying.\n\nResearchers introduced what they call a \"capability slice\" — a group of evaluation samples defined by shared background condition, task type, operation, and output constraint. The idea is to bridge a persistent gap: benchmark scores tell you a model failed, but not which part of your training data to fix. The team built an evaluation taxonomy, a data taxonomy, and mapping rules that connect the two, creating a closed loop from observed failure to targeted intervention. Two case studies tested the system in opposite directions to stress-test the approach.\n\nThe numbers make the stakes concrete. In the first case, continued pre-training dropped BBH scores by -46.82%, which looked like a reasoning collapse — but the loop traced the culprit to a single masked end-of-sequence token loss, not bad data. Fixing the token restored BBH to 66.44, above the original baseline, with no data changes at all. In the second, a math-reasoning weakness was decomposed by solving operation, and targeted sampling lifted AIME 2025 and AIME 2026 Pass@128 from 6.67 and 0.00 respectively to 26.67 each.\n\nThe reason this matters: training LLMs today still depends heavily on engineers developing intuition about what data to add or cut. A reproducible, auditable loop that reaches the right verdict in opposite-direction cases is a meaningful step toward making that process systematic rather than artisanal.\n\nThe framework is not a product — it is a research loop on paper. Whether it scales past two controlled case studies to the messy reality of frontier training runs is the question labs will want answered before they hand over the diagnostic keys.","[\"ai\",\"llm\",\"machine-learning\",\"benchmarks\"]","2026-06-30T04:00:00.000Z","2026-06-30T05:02:28.969Z","2026-06-30T05:02:36.947Z","published",null,[24],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"The body states AIME 2025 and 2026 Pass@128 started from 'near-zero' but the source gives exact figures (6.67 and 0.00 respectively), and the BBH case omits the specific −46.82% drop figure that would give readers concrete stakes — replace vague characterizations with the actual numbers from the source.","resolved","https:\u002F\u002Fcdn.xyz.onl\u002Farticle-images\u002Fa-closed-loop-for-turning-benchmark-failures-into-data-fixes.webp","ai",[31,33,34,35],"llm","machine-learning","benchmarks",[37],{"name":38,"url":39},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.28471",0,{"sections":42},[43,47,52,57,62,67,72,77,82,87,92,96,101,106],{"name":44,"slug":31,"count":45,"latest_published_at":46},"AI",2590,"2026-07-16T04:00:00.000Z",{"name":48,"slug":49,"count":50,"latest_published_at":51},"Security","security",294,"2026-07-15T19:59:48.000Z",{"name":53,"slug":54,"count":55,"latest_published_at":56},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":58,"slug":59,"count":60,"latest_published_at":61},"Policy","policy",158,"2026-07-16T00:02:48.000Z",{"name":63,"slug":64,"count":65,"latest_published_at":66},"Hardware","hardware",122,"2026-07-14T19:46:26.000Z",{"name":68,"slug":69,"count":70,"latest_published_at":71},"Consumer Tech","consumer-tech",93,"2026-07-13T13:20:48.000Z",{"name":73,"slug":74,"count":75,"latest_published_at":76},"Software","software",70,"2026-07-13T19:52:25.000Z",{"name":78,"slug":79,"count":80,"latest_published_at":81},"Science","science",66,"2026-07-10T10:29:37.000Z",{"name":83,"slug":84,"count":85,"latest_published_at":86},"Dev Tools","dev-tools",59,"2026-07-07T04:00:00.000Z",{"name":88,"slug":89,"count":90,"latest_published_at":91},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":93,"slug":94,"count":90,"latest_published_at":95},"Startups","startups","2026-06-29T20:55:50.000Z",{"name":97,"slug":98,"count":99,"latest_published_at":100},"General","general",29,"2026-07-10T22:28:58.000Z",{"name":102,"slug":103,"count":104,"latest_published_at":105},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":107,"slug":108,"count":109,"latest_published_at":110},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]