[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-new-kv-cache-design-boosts-multi-agent-llm-throughput-up-to-2x":10,"sections":35},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":30,"feedback":34,"feedback_at":22,"cost_usd":34,"total_tokens":34},5386,"new-kv-cache-design-boosts-multi-agent-llm-throughput-up-to-2x","New KV Cache Design Boosts Multi Agent LLM Throughput Up to 2x","A new paging system splits AI model caches into a contiguous pool for shared prompts and a granular pool for replies, lifting throughput up to 2x.","A new caching layer called GraniKV could nearly double how many requests an AI server can handle when several AI agents share the same prompt.\n\nMost AI inference engines store a request's context (the running memory of a conversation, known as the KV cache) using one fixed chunk size for everything. That works fine for a single user, but multi-agent systems often have many requests sharing one long prefix, like a system prompt or a list of tools, while each request's own reply differs. GraniKV splits the two, keeping the shared prefix in one contiguous block and each request's unique suffix in small, token-level chunks, then picks whichever processing backend fits the bottleneck at each step. In tests on Llama-3.1-8B, Qwen-2.5-14B, and Qwen-2.5-32B, the researchers report throughput gains of 2.16x, 1.98x, and 1.57x over a standard paged-serving baseline with a 16,000-token shared prefix.\n\nThe more revealing number is what happens when agents do not share identical prompts, the realistic case for most multi-agent deployments. There, a rival trick the researchers also tested, batching identical prefixes for shared computation, stops helping at all, while GraniKV still delivers a 1.95x speedup. That suggests the real win here is not clever attention math but simply refusing to force two very different kinds of data into the same storage format.\n\nThis is an arXiv preprint measured against an unnamed baseline, not an independently reproduced result, so read the numbers as promising rather than settled.","[\"kv cache\",\"llm inference\",\"multi-agent systems\",\"ai infrastructure\"]","2026-08-18T04:00:00.000Z","2026-08-18T18:06:04.882Z","2026-08-18T18:06:17.105Z","published",null,[],"ai",[26,27,28,29],"kv cache","llm inference","multi-agent systems","ai infrastructure",[31],{"name":32,"url":33},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.15584",0,{"sections":36},[37,41,45,50,55,60,65,70,75,79,84,89,94,99],{"name":38,"slug":24,"count":39,"latest_published_at":40},"AI",3293,"2026-08-20T04:00:00.000Z",{"name":42,"slug":43,"count":44,"latest_published_at":40},"Security","security",435,{"name":46,"slug":47,"count":48,"latest_published_at":49},"Policy","policy",210,"2026-08-19T09:32:27.000Z",{"name":51,"slug":52,"count":53,"latest_published_at":54},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":56,"slug":57,"count":58,"latest_published_at":59},"Hardware","hardware",140,"2026-08-19T18:25:42.000Z",{"name":61,"slug":62,"count":63,"latest_published_at":64},"Consumer Tech","consumer-tech",95,"2026-08-18T16:05:00.000Z",{"name":66,"slug":67,"count":68,"latest_published_at":69},"Science","science",90,"2026-08-19T18:41:02.000Z",{"name":71,"slug":72,"count":73,"latest_published_at":74},"Software","software",73,"2026-08-18T07:51:50.000Z",{"name":76,"slug":77,"count":78,"latest_published_at":18},"Dev Tools","dev-tools",69,{"name":80,"slug":81,"count":82,"latest_published_at":83},"Startups","startups",47,"2026-08-19T19:13:46.000Z",{"name":85,"slug":86,"count":87,"latest_published_at":88},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":90,"slug":91,"count":92,"latest_published_at":93},"General","general",33,"2026-08-18T22:18:13.000Z",{"name":95,"slug":96,"count":97,"latest_published_at":98},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":100,"slug":101,"count":102,"latest_published_at":103},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]