Amazon has a new tool for watching AI agents think, and it lives in your IDE instead of a browser tab.
CloudWatch Omni is a new observability, evaluation, and experimentation product for AI agents, shipping as an extension for VS Code and Kiro plus a standalone web app kept separate from the AWS Management Console. It captures full traces of every LLM call, tool invocation, and reasoning step an agent takes, then runs 17 built-in evaluators scoring things like coherence, faithfulness, and routing correctness. Developers can compare prompt versions side by side in a playground, build test datasets from production traffic, and run experiments across configurations to catch regressions automatically. A Cloud Login feature makes syncing to AWS optional, so teams can work entirely locally before deciding to send telemetry to CloudWatch.
Standard uptime and latency metrics don't tell you whether an agent's answer was actually correct, or why it took an odd path, which has forced teams to hand-review logs across scattered dashboards. Omni's bet is that fixing agent reliability requires treating quality evaluation as a first-class, in-the-loop step, not an afterthought bolted onto conventional monitoring. That's also a direct pitch against the growing pile of standalone LLM-ops startups doing the same job.
AWS folding this into CloudWatch, its two-decade-old monitoring product, is a reminder that observability is being redefined feature by feature as AI agents arrive, not because the old definition failed, but because it was never built to grade an agent's reasoning.