Dev Tools/ llm-agents · testing · ci-cd · regression-testing

Chronicle Turns Recorded Agent Failures Into CI Tests

A new record-and-replay tool called Chronicle lets teams replay a single recorded LLM agent failure as a repeatable regression test without burning model calls.

LLM agents fail in ways you can't reproduce, and now there's a tool built to fix that.

Chronicle records an agent run at its non-deterministic boundaries - model calls, tool outputs, anything that won't repeat the same way twice - and saves them as immutable envelopes. Its key trick is "cut-point replay": you replay most of a recorded run from the log, but let a chosen slice execute live against new code. That turns a one-off production incident into a regression test that runs in CI. On a benchmark of 6 recorded failures, recording overhead was 23 microseconds per crossing, and full replay made zero live model calls while staying bit-identical across 20 runs. The tests correctly failed on faulty code and passed on fixed code for all 6 cases.

This matters because agent debugging today mostly means staring at a trace and hoping the bug shows up again. Existing record-and-replay tools capture runs to score or inspect them, not to gate a code change - Chronicle is explicitly built for the CI use case, which is the missing piece between "we saw it fail once" and "we can stop it from failing again." The mutation-testing result is the more interesting data point: cut-point tests caught every mutant that let an unsafe tool action through, while a baseline that just stubs every boundary caught none, using the identical assertion.

That gap - not the latency numbers - is the real finding: stubbing everything gives you a test that always passes, which is worse than no test at all. Whether Chronicle's approach holds up outside a 6-incident benchmark and simulated model boundaries is the open question worth watching.

TR

The Revision

Written by an AI system from the public sources credited above. How we write →