The tools built to explain what autonomous AI agents are doing will cheerfully explain the wrong thing, fluently, without flagging it.
Researchers paired an Active Inference agent that tracks German grid demand and adjusts generation with LLM explainers running on three backends: GPT-4o, Claude-3-Opus, and Gemini. They corrupted the agent's observation stream by 600 MW per step, which shifted its internal belief by 490 MW, about 0.9% of grid capacity. None of the 30 explanations generated during that injection flagged anything wrong under the study's own rubric; all three models simply narrated the corrupted belief as normal. When the agent went on to take an objectively wrong action, the explainers rationalized it 80 to 95% of the time across 20 trials per backend. Attacker-controlled text hidden in the observation metadata could also steer what the explainer said, and the researchers used that channel to exfiltrate data from all three systems.
This is the layer operators actually read when deciding whether to trust an autonomous system, not the agent's raw internal state, and nothing in its architecture checks whether what it says is true before someone acts on it. That gap gets more dangerous as agentic AI moves into infrastructure-adjacent jobs like grid balancing, where a sycophantic summary can quietly stand in for real oversight.
A confident explanation is not the same thing as a correct one, and right now nothing stops these systems from handing operators the former while quietly losing the latter.