[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-a-new-framework-ranks-452-llm-benchmarks-for-banking-work":10,"sections":40},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":30,"tags":31,"sources":35,"feedback":39,"feedback_at":22,"cost_usd":39,"total_tokens":39},3402,"a-new-framework-ranks-452-llm-benchmarks-for-banking-work","A New Framework Ranks 452 LLM Benchmarks for Banking Work","Researchers mapped 452 public LLM benchmarks onto 41 work activities and 38 banking domains, then used a weighted Elo system to score models for finance work.","Picking the right LLM for a bank just got a methodology.\n\nResearchers published a meta-benchmarking framework that organizes 452 publicly reported LLM benchmarks into 41 O*NET Generalized Work Activities, then rolls those up into 38 banking business domains drawn from the BIAN standard. A multiplicative weighting scheme, scoring each benchmark on discrimination, coverage, and recency, feeds a pairwise Elo tournament that produces work-activity scores without requiring raw score normalization. The framework was tested on a June 2026 snapshot covering 288 models from 25 organizations. The full methodology, taxonomy, and limitations are documented to let other institutions reproduce it.\n\nPublic leaderboards like MMLU-Pro rank models on global averages that say little about whether a model can handle compliance reasoning or multi-turn customer service. This framework argues that a model topping a coding benchmark might be the wrong pick for loan origination or risk management, and gives procurement teams a structured way to make that distinction.\n\nThe banking sector has been slow to formalize LLM governance; a reproducible, publicly documented scoring system could push that conversation forward, though its value ultimately depends on which benchmarks institutions choose to feed it.","[\"ai\",\"finance\",\"benchmarks\",\"llm-evaluation\"]","2026-07-03T04:00:00.000Z","2026-07-03T05:07:30.105Z","2026-07-03T05:07:32.834Z","published",null,[24],{"id":25,"reviewer":26,"round":27,"reason":28,"status":29},"editor-r1","editor",1,"The dek states '38 banking domains' but the body opens by correctly saying '38 banking domains' yet also states '41 generalized work activities' — however the body contradicts itself by saying the benchmarks are mapped onto '41 generalized work activities and then rolls those up into 38 banking business domains,' which is accurate, but the headline says '452 benchmarks' while the dek omits the 41 work-activity layer entirely, making the methodology description in the dek incomplete and potential","resolved","ai",[30,32,33,34],"finance","benchmarks","llm-evaluation",[36],{"name":37,"url":38},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2607.01740",0,{"sections":41},[42,46,51,56,61,66,71,76,81,86,91,95,100,105],{"name":43,"slug":30,"count":44,"latest_published_at":45},"AI",2590,"2026-07-16T04:00:00.000Z",{"name":47,"slug":48,"count":49,"latest_published_at":50},"Security","security",294,"2026-07-15T19:59:48.000Z",{"name":52,"slug":53,"count":54,"latest_published_at":55},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":57,"slug":58,"count":59,"latest_published_at":60},"Policy","policy",158,"2026-07-16T00:02:48.000Z",{"name":62,"slug":63,"count":64,"latest_published_at":65},"Hardware","hardware",122,"2026-07-14T19:46:26.000Z",{"name":67,"slug":68,"count":69,"latest_published_at":70},"Consumer Tech","consumer-tech",93,"2026-07-13T13:20:48.000Z",{"name":72,"slug":73,"count":74,"latest_published_at":75},"Software","software",70,"2026-07-13T19:52:25.000Z",{"name":77,"slug":78,"count":79,"latest_published_at":80},"Science","science",66,"2026-07-10T10:29:37.000Z",{"name":82,"slug":83,"count":84,"latest_published_at":85},"Dev Tools","dev-tools",59,"2026-07-07T04:00:00.000Z",{"name":87,"slug":88,"count":89,"latest_published_at":90},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":92,"slug":93,"count":89,"latest_published_at":94},"Startups","startups","2026-06-29T20:55:50.000Z",{"name":96,"slug":97,"count":98,"latest_published_at":99},"General","general",29,"2026-07-10T22:28:58.000Z",{"name":101,"slug":102,"count":103,"latest_published_at":104},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":106,"slug":107,"count":108,"latest_published_at":109},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]