AI agents that use tools often sit idle while waiting for a webpage to load or a function to return - and a new paper argues that dead time is wasted compute.
Researchers propose Second Thought, a training-free inference framework for LLM agents that follow the ReAct pattern of alternating reasoning, acting, and observing. The moment an agent finishes a Thought phase and dispatches an action, Second Thought forks four auxiliary reasoning branches and decodes them in parallel while the action executes and the environment responds. When the observation finally arrives, those extra thoughts get merged back into the main reasoning thread. Tested across three agentic benchmarks and three reasoning LLMs - nine model-benchmark combinations in total - the method cut the average number of turns needed in every single pairing, and reduced main-thread decoding by up to 43 percent in six of them. Accuracy barely moved in most cases, and two combinations saw solid gains of 12.4 and 10.2 percentage points.
The real test is against a compute-matched control that pours the same extra budget into the main thread's own reasoning instead of parallel branches. Second Thought still wins there, hitting higher accuracy with 1.3 to 3.2 times less sequential decoding. That is the more interesting claim than raw speed - it suggests idle time, not just more compute, was the bottleneck.
It is a training-free trick sitting on top of existing models, which makes it easy to bolt onto current agent stacks - but the gains ride on how long real tool calls and API round-trips actually take, and this is a fresh arXiv preprint, not a benchmark run in production.