Researchers found that small language models can meaningfully guide reinforcement learning agents, but only if you give them something worth reasoning about.
The paper introduces ASK+, a method for selectively querying a small language model (SLM) when a reinforcement learning agent is uncertain about its next move. Earlier approaches — called vanilla uncertainty-gated methods — almost never let the SLM contribute at all, because the prompt handed to it was too bare to support genuine reasoning. ASK+ fixes this by feeding the SLM a trajectory-aware context: a partially revealed map, a history of visited positions, and a log of past actions. It also adds structured chain-of-thought prompting to push the model toward deliberate inference rather than reflex. The result is an SLM that occasionally overrides the policy with something better.
The benchmark numbers are not subtle. On the FourRooms navigation task, success rates climbed from 53% with the vanilla approach to 70% with ASK+. On DoorKey, the gain was narrower — 89% to 93% — but still real. Perhaps more striking: a 2-billion-parameter model matched or beat a 4-billion-parameter one across all test environments, suggesting that prompt design is doing work that model scale usually gets credit for.
This lands at a moment when the AI field is still largely chasing scale as the default lever for performance. If the bottleneck in agentic settings turns out to be context quality rather than model size, that changes the economics of deploying AI in constrained or embedded environments — though a handful of grid-world benchmarks is a long way from a production robot or autonomous vehicle.