[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"branding":3,"analytics":7,"article-researchers-fix-a-memory-bottleneck-in-hybrid-ai-models":10,"sections":35},{"siteName":4,"siteTagline":5,"publisherName":4,"contactEmail":6},"The Revision","Tech news, decoded.","editor@therevision.news",{"gaMeasurementId":8,"adsenseClientId":9},"G-ZW2MV82GYR","ca-pub-8533917693782264",{"article":11},{"id":12,"slug":13,"title":14,"dek":15,"body_md":16,"tags_json":17,"published_at":18,"created_at":19,"updated_at":20,"status":21,"review_note":22,"review_notes":23,"image_url":22,"persona_id":22,"persona_name":22,"section":24,"tags":25,"sources":30,"feedback":34,"feedback_at":22,"cost_usd":34,"total_tokens":34},5814,"researchers-fix-a-memory-bottleneck-in-hybrid-ai-models","Researchers Fix a Memory Bottleneck in Hybrid AI Models","A new method called TreeWY frees up memory in speculative decoding for hybrid AI models by eliminating costly per-draft state snapshots.","A new algorithm called TreeWY fixes a memory bottleneck that was quietly undermining one of the industry's favorite speed tricks for hybrid AI models.\n\nMost efficient open models today mix transformer layers with Gated DeltaNet layers, which use linear attention and a small fixed-size recurrent state instead of a growing key-value cache. That's great for ordinary decoding, but it breaks speculative decoding, the technique where a model drafts several tokens ahead and verifies them in a batch. Verifying and rolling back drafts normally requires snapshotting the full recurrent state at every draft position, and those snapshots can't be shared across branches of a draft tree, so a wide tree quickly runs out of memory. TreeWY removes the snapshots entirely, using a tree-structured transform of the gated delta rule to compute every draft node's output with one triangular solve and reconstruct only the single accepted state when a branch is committed, storing a small pseudo-value matrix instead of per-node states.\n\nIn tests on Qwen3.5 at 35B and 397B parameters, the method cut recurrent-state and KV-cache memory pressure at the same acceptance length, which translated into higher throughput and notably lower time-to-first-token wherever memory was the constraint. Where memory wasn't the bottleneck, the cost was a few percent. The freed memory also makes wider, higher-acceptance draft trees affordable for the first time, though the researchers note that doesn't yet translate into a throughput win.\n\nIt's a plumbing fix, not a new architecture, and that's exactly why it matters: as more open models go hybrid to save memory, the industry needs speculative decoding to keep working, or the speed gains from smaller models get eaten by verification overhead. This is a preprint tested on one model family, so treat the numbers as promising rather than proven.","[\"speculative-decoding\",\"hybrid-models\",\"gated-deltanet\",\"inference-optimization\"]","2026-08-24T04:00:00.000Z","2026-08-24T05:08:12.652Z","2026-08-24T05:08:24.553Z","published",null,[],"ai",[26,27,28,29],"speculative-decoding","hybrid-models","gated-deltanet","inference-optimization",[31],{"name":32,"url":33},"arXiv cs.AI","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.20961",0,{"sections":36},[37,41,45,50,55,60,65,70,75,80,85,90,95,100],{"name":38,"slug":24,"count":39,"latest_published_at":40},"AI",3325,"2026-08-24T09:09:31.000Z",{"name":42,"slug":43,"count":44,"latest_published_at":18},"Security","security",461,{"name":46,"slug":47,"count":48,"latest_published_at":49},"Policy","policy",218,"2026-08-23T19:30:00.000Z",{"name":51,"slug":52,"count":53,"latest_published_at":54},"Deals","deals",179,"2026-06-29T20:02:07.000Z",{"name":56,"slug":57,"count":58,"latest_published_at":59},"Hardware","hardware",145,"2026-08-22T21:25:33.000Z",{"name":61,"slug":62,"count":63,"latest_published_at":64},"Consumer Tech","consumer-tech",95,"2026-08-18T16:05:00.000Z",{"name":66,"slug":67,"count":68,"latest_published_at":69},"Science","science",91,"2026-08-20T10:01:48.000Z",{"name":71,"slug":72,"count":73,"latest_published_at":74},"Software","software",73,"2026-08-18T07:51:50.000Z",{"name":76,"slug":77,"count":78,"latest_published_at":79},"Dev Tools","dev-tools",69,"2026-08-18T04:00:00.000Z",{"name":81,"slug":82,"count":83,"latest_published_at":84},"Startups","startups",50,"2026-08-22T16:23:09.000Z",{"name":86,"slug":87,"count":88,"latest_published_at":89},"Gaming","gaming",41,"2026-07-09T04:00:00.000Z",{"name":91,"slug":92,"count":93,"latest_published_at":94},"General","general",33,"2026-08-18T22:18:13.000Z",{"name":96,"slug":97,"count":98,"latest_published_at":99},"Reviews","reviews",20,"2026-06-24T12:00:01.000Z",{"name":101,"slug":102,"count":103,"latest_published_at":104},"How-To","how-to",6,"2026-06-16T09:00:00.000Z"]