AI/ llm-agents · inference-cost · webassembly · ai-research

Auto Compiles LLM Agent Runs Into Cheap, Verified Programs

A new arXiv paper describes Auto, a compiler that cuts LLM agent costs 6.4x end-to-end, and quantifies exactly how it fails silently when calibration slips.

A paper posted to arXiv proposes treating LLM agent behavior like compilable source code, so the predictable parts never have to hit a frontier model twice.

Researchers at RightNow AI built a system called Auto that watches a frontier model agent work, identifies which steps always produce the same output, and packages those as cognition binaries: WebAssembly artifacts whose manifests carry measured guarantees and whose declared capabilities are enforced by a sandbox. When a compiled binary encounters an unfamiliar input, it falls back to the full model, captures the new trace, and recompiles. On AUTO-BENCH, a benchmark the team introduces and pre-registers in the arXiv paper, 87.1% of 560 recorded frontier-agent spans are witnessed-deterministic (three of the four task families hit 100.0%); on a 300-item stream with three scheduled distribution shifts, Auto cut marginal cost from 59 to 2 micro-dollars per item, a 6.4x reduction end-to-end, at 96.9% parity on witnessed inputs. Code is on GitHub.

The paper's most useful contribution may be the failure taxonomy. A loose guard silently mislabeled 48.9% of compiled answers: costs drop, wrong answers go out, and nothing alerts you. The authors conclude that calibration and reference fidelity, not model capability, decide whether cheaper also means correct.

For repetitive, high-volume workloads the cost reduction is real on paper; the 48.9% silent failure rate is the figure that will determine whether it stays real in production.

TR

The Revision

Written by an AI system from the public sources credited above. How we write →