A new audit method can catch hosted LLM APIs serving a degraded version of the model they advertise, without needing the provider to hand over any probability data.
Researchers describe Ventor-QTest in a new arXiv paper: a black-box auditing framework built around two components. The first repeatedly sends the same frozen prompt to a target API, tallies the returned text, and reconstructs an output distribution to calculate what the paper calls average fidelity loss (AFL). The second runs longer, independent sequences and examines the extreme tail of a run-level surprisal statistic to compute extreme fidelity loss (EFL). Tested against three routes where logprobs were available for comparison, AFL tracked closely with a logprob-based reference measure, and across seven route snapshots, EFL varied meaningfully from route to route.
Neither AFL nor EFL correlated much with accuracy on GPQA-Diamond, a static knowledge benchmark. EFL did track a drop in Terminal-Bench pass rates as tasks grew longer, the kind of degradation that compounds quietly in multi-step agent workflows rather than surfacing on a single-turn quiz. That distinction matters as more products chain LLM calls into agents, where a provider quietly swapping in a cheaper or quantized model could hurt real-world reliability without ever showing up on a benchmark leaderboard.
The code is open-source, released through Tencent's AI-Infra-Guard project, but the paper does not name which vendors or routes were actually audited. It is a measurement tool, not yet a public accountability report; someone still has to point it at specific providers and publish what they find.