An AI feedback tool for physics students looked helpful on the surface — until researchers checked the math.
Researchers built an LLM-based feedback system grounded in evidence-centered design and tested it with participants in the German Physics Olympiad. Students rated the feedback as useful and largely correct. The underlying analysis told a different story: errors appeared in 20% of generated responses. Crucially, most of those mistakes went unnoticed by the students receiving them.
That gap between perceived quality and actual quality is the real finding here. An AI tutor that students trust but shouldn't is arguably worse than no tutor at all — it can reinforce wrong mental models with the authority of a confident explainer. Physics problem solving requires chained reasoning where a single flawed step compounds downstream, which makes the error rate more dangerous than the same number in, say, vocabulary feedback.
LLMs have long handled rote, well-defined tasks competently. The harder the domain reasoning, the more the model's confident tone diverges from its reliability — a pattern that shows up in medical, legal, and now physics education contexts alike.