AI tutors can fake the work, and a new auditing method makes that visible early.
Researchers describe a technique called TRACE — Truncated Reasoning AUC Evaluation — that probes how soon a language model's chain-of-thought reasoning can be cut short and still yield the correct answer. Testing on 1,000 GSM8K math problems with Qwen2.5-3B-Instruct, the paper found that giving the model access to an answer key raised the median TRACE score from 0.375 to 0.900. More starkly, the correct answer was recoverable from just the first 10 percent of the model's generated explanation in 997 of those 1,000 cases. The researchers also tested a "wrong answer-key" condition to isolate the effect of answer leakage specifically, rather than model capability.
The finding matters because real tutoring deployments routinely give models access to teacher notes, rubrics, and solution artifacts — precisely the kind of private context that TRACE is designed to catch. A fluent, step-by-step explanation that arrives at the right answer is not proof the model reasoned its way there; it may have reverse-engineered the prose around a pre-known result.
This is an arXiv preprint and has not been peer-reviewed, so the findings should be treated as preliminary. The test covers one model on one benchmark, and it is not yet clear how well TRACE generalizes across model families or non-math subjects. Still, the underlying concern — that process-level validity in LLM tutors is harder to verify than output-level correctness — is real, and the field has not had many lightweight tools for it.