A new framework scores AI-generated math problems by grading the reasoning behind them, not just the words on the page.
The system, called ProIQA (Process-based Item Quality Assessment), comes from a research paper published this week. It targets a gap in automatic item generation, where AI tools produce math questions for platforms like adaptive tutoring apps, but there has been no scalable way to check if those questions are actually good. Today that mostly means manual review, which does not scale, or shallow metrics based on how a problem's stem is worded, which miss whether the problem tests the reasoning it claims to. ProIQA instead has large language models build a reasoning tree from a problem's solution, then uses a graph neural network to encode how the steps in that solution depend on each other. That process-level representation is fused with the problem's stem text so the system can judge knowledge concepts, difficulty, and disciplinary competencies together, and the researchers tested it on K12 math datasets.
That process-aware layer is the real shift. Most automatic item generation quality checks today amount to skimming a question's wording or checking it against a difficulty label a human assigned once. Grading the solving process instead means catching problems that look fine on the surface but do not actually require the reasoning they are supposed to test, the kind of flaw that quietly degrades a personalized learning app's item bank.
The paper is a lab benchmark, not a shipped product, so the real test is whether ProIQA holds up on item banks nobody has carefully curated for a research dataset.