LLM agents spend a surprising fraction of their time just waiting — and a new open-source controller aims to claw that time back.
When an AI agent calls an external tool, the GPU sits idle until the result returns. That idle gap eats 16-37% of wall time in the SPORK researchers' own workloads and 35-61% in prior work. SPORK — short for Self-sPeculative fORKing — addresses this by forking a lightweight probe at generation start to predict which tool the model is about to call. If the probe is confident enough, SPORK dispatches that tool call early, overlapping execution with the remaining chain-of-thought decoding. On acceptance, the result is waiting when reasoning finishes; on a wrong guess, it falls back to normal serial execution with no impact on correctness.
The practical gains are real: on the GAIA benchmark, SPORK cut Qwen3-32B's 95th-percentile latency from 131.9 seconds to 108.1 seconds — an 18% reduction. The probe predicted the correct tool name with 74.6-99.6% accuracy across five benchmarks, the system works across model sizes from 4B to 32B parameters, and it requires no retraining, no historical traces, and no auxiliary models.
Speculative execution is not a new idea — token-level speculative decoding has been in use for some time — but applying it at the tool-call level without any training overhead is a genuine gap SPORK fills. The bigger question is adoption: the approach deploys as a thin wrapper over standard completion APIs, which lowers the bar considerably, but agentic inference stacks vary wildly in how much the operator actually controls the generation loop.