A research framework called ProvenanceGuard wants to stop AI agents from doing things their users never asked for.
As large language models gain access to external tools — running code, querying databases, sending messages — the risk grows that an agent will invoke a tool in ways the user never intended. Researchers describe this as "misalignment" and note it can produce consequences that are hard to undo. Existing safeguards typically use a second LLM to judge whether an action is appropriate, but that approach produces inconsistent verdicts and leaves little audit trail. ProvenanceGuard takes a different approach: before any tool runs, it checks whether the proposed action is traceable back to the user's original query through a multi-stage pipeline that tests for three categories of misalignment.
The performance gap over the baseline is hard to dismiss. On the Agent-SafetyBench benchmark, the error rate on misaligned traces dropped from 42.9% to 1.8%; on WorkBench it fell from 32.1% to 17.3%. Crucially, the system also reduced unnecessary interventions on legitimate actions, cutting that burden from 30.5% to 12.8% — meaning it gets in the way less while blocking more of the bad stuff.
The LLM-as-a-judge pattern has become the default guardrail in agentic pipelines partly because it requires almost no infrastructure — just another model call. ProvenanceGuard suggests that a structured reasoning layer can do the job far more reliably, though whether that improvement holds when agents operate across dozens of tools in messy real-world environments is a question the benchmarks can't fully answer yet.