A new benchmark wants to stop grading AI trading agents on luck.
Researchers introduced CLQT, a closed-loop evaluation framework for large language model agents acting as portfolio managers. Unlike existing benchmarks that rank agents by raw returns over a fixed period, CLQT is designed to diagnose where an agent's reasoning breaks down. Each agent runs a five-stage cycle — gather, synthesize, allocate, execute, reflect — and every decision round is sealed into a hash chain so results can be independently reconstructed. The environment accounts for transaction costs, financing costs, and strategy consistency, and uses a time gate to prevent look-ahead data leakage, a known source of inflated results in prior work.
The problem CLQT is solving is real: a strong return in a backtest can be driven almost entirely by market conditions rather than any skill the agent actually has. By separating outcome from process, the benchmark produces a five-axis scorecard — Coherence, Acuity, Composure, Discipline, and Reliability — that maps what an agent can and cannot do rather than just where it ranked on a leaderboard. Coherence is evaluated partly by a separate, held-out LLM to reduce self-grading bias, which is a small but notable methodological detail.
The researchers validated CLQT on a contamination-controlled multi-model backtest and a live broker track on post-cutoff data. Whether the financial industry adopts a framework built by academics for testing agents it may not fully trust is a separate question — but the benchmark at least asks the right one.