[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-a-better-way-to-measure-how-compression-changes-ai-outputs":10,"sections":34},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":29,"feedback":33,"feedback_at":22,"cost_usd":33,"total_tokens":33},9660,"a-better-way-to-measure-how-compression-changes-ai-outputs","A Better Way to Measure How Compression Changes AI Outputs","Researchers find that total variation, not the widely used KL divergence, accurately predicts how often compressed language models flip their answers.","A new study says the metric everyone quotes for compressed AI models measures the wrong thing.\n\nResearchers tested 802 compressed and perturbed versions of 19 open-source language models across five text corpora and nine unrelated perturbation methods. They measured how often a compressed model's top-choice word differed from the original dense model's choice, a measure they call the flip rate. A simpler statistic called total variation predicted that flip rate almost one-to-one, with a median ratio of 1.05. KL divergence, the metric compression reports actually use, connects to flip rate only loosely, through a square root and a correction factor that varies fourfold across models and corpora.\n\nThat gap is not academic: when comparing two compressors whose flip rates differ by 10 percent or more, KL divergence picks the wrong one as 'more faithful' 11 percent of the time, versus 1 percent for total variation. The pattern held up in follow-up tests on a held-out code corpus and on new models with real compression kernels. For anyone deploying a quantized or pruned model and expecting it to act like the original, the KL number in a vendor's report may be quietly underselling how much actually changed, including in speculative decoding setups like vLLM, where total variation predicted draft-acceptance rates within 1 to 2 percent.\n\nCompression benchmarks already lean on numbers that flatter the compressor; this is one more reason to check which distance metric is actually behind a 'minimal divergence' claim.","[\"ai\",\"model-compression\",\"research\",\"llms\"]","2026-10-02T04:00:00.000Z","2026-10-03T05:01:31.299Z","2026-10-03T05:01:37.514Z","published",null,[],"ai",[24,26,27,28],"model-compression","research","llms",[30],{"name":31,"url":32},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.00694",0,{"sections":35},[36,39,43,47,52,56,60,65,70,75,80,85,90,95],{"name":37,"slug":24,"count":38,"latest_published_at":18},"AI",5976,{"name":40,"slug":41,"count":42,"latest_published_at":18},"Security","security",842,{"name":44,"slug":45,"count":46,"latest_published_at":18},"Policy","policy",438,{"name":48,"slug":49,"count":50,"latest_published_at":51},"Deals","deals",317,"2026-10-01T22:00:00.000Z",{"name":53,"slug":54,"count":55,"latest_published_at":18},"Hardware","hardware",199,{"name":57,"slug":58,"count":59,"latest_published_at":18},"Science","science",173,{"name":61,"slug":62,"count":63,"latest_published_at":64},"Consumer Tech","consumer-tech",155,"2026-10-01T19:54:10.000Z",{"name":66,"slug":67,"count":68,"latest_published_at":69},"Dev Tools","dev-tools",96,"2026-10-01T16:57:03.000Z",{"name":71,"slug":72,"count":73,"latest_published_at":74},"Software","software",93,"2026-09-30T21:41:11.000Z",{"name":76,"slug":77,"count":78,"latest_published_at":79},"Startups","startups",90,"2026-10-01T21:55:22.000Z",{"name":81,"slug":82,"count":83,"latest_published_at":84},"Gaming","gaming",53,"2026-10-02T02:50:39.000Z",{"name":86,"slug":87,"count":88,"latest_published_at":89},"General","general",50,"2026-09-30T21:37:54.000Z",{"name":91,"slug":92,"count":93,"latest_published_at":94},"Reviews","reviews",31,"2026-09-28T14:31:34.000Z",{"name":96,"slug":97,"count":98,"latest_published_at":99},"How-To","how-to",7,"2026-10-01T09:00:00.000Z"]