A new serving technique lets AI agents start computing likely next steps before they know which branch they'll take.
Agentic LLM workflows constantly hit branch points: should the agent call this tool or that one, follow this reasoning path or another? The system can't start the next step until that branch resolves, even if the answer was predictable or has already been computed before. Caching does not help, because the cache key needed to look up a reusable result is not known until the branch resolves. Researchers propose DynBranch, which assigns an unresolved branch a stable coordinate so it can be addressed before it resolves. That lets candidate subgraphs run speculatively during resolution, and lets completed results get reused across later requests, with a two-level controller deciding when the expected benefit is worth the extra compute. It slots in at the model-API boundary, so it works without changes to agent harnesses or model engines. In tests across four agentic workloads on Qwen3-32B with four H200 GPUs, DynBranch cut mean latency by up to 32% compared with each workload's strongest prior system, and by 46-66% compared with a baseline that does no reuse at all. The gains held on a smaller Qwen3-8B model running on a single consumer RTX 4090.
Agentic pipelines chain many LLM calls together, so per-step latency compounds fast, and most of the industry's fixes so far have targeted better models or bigger GPUs. DynBranch is a plumbing fix instead: it treats the agent's decision tree as something to pre-compute around rather than wait on, without touching the model itself. That kind of infrastructure-level speedup is easier to adopt than a model swap, since it works underneath whatever agent framework is already in place.
Worth noting: the headline 32% figure is measured against each workload's strongest prior system, not against doing nothing. The flashier 46-66% number is versus a no-reuse floor that nobody was actually shipping. How this holds up on messier, real-world agent pipelines is still an open question.