AI agents are getting better at calling functions — and better at confidently getting those calls wrong.
Researchers have published a paper arguing that standard benchmarks for function-calling agents are quietly broken. The metrics reward models for making a call — any call — even when the right answer is "I don't know." That incentive structure pushes models toward confident hallucinations, which is mostly harmless in a chatbot and potentially catastrophic when the agent is wiring a bank transfer or adjusting a medical device. The proposed fix is a trainable filter that sits in front of the model's function-calling output, estimates uncertainty, and suppresses calls that look shaky — without touching the underlying model weights.
The "don't touch the model" part matters more than it might seem. Most safety interventions require retraining or fine-tuning the base model, which is expensive and often regressive. A lightweight, bolt-on filter is something an enterprise team can actually deploy on a model they didn't build and can't retrain. That's the realistic production path for most companies running agents on top of third-party APIs.
This sits in a growing pile of research acknowledging that benchmark performance and real-world reliability are different problems — a gap the industry has spent years papering over with bigger models rather than better guardrails.