A new arXiv preprint puts six large language models to the test on a narrow but useful job: turning a game tester's plain-English bug report into a formal PDDL planning goal.
The paper (arXiv:2609.18731) built a prompt template through iterative testing, then measured how well GPT-4.1, Gemini 2.5 Flash, and four other contemporary models could convert real tester language into valid PDDL, the formal syntax classical planners use to define goals and states. All six models got it right more than 92% of the time. Gemini 2.5 Flash led on accuracy at 96% and had the fewest false positives, while GPT-4.1 answered fastest. Most of the remaining errors traced back to ambiguous phrasing from testers or gaps in how a given game's domain was represented in PDDL.
That distinction matters because PDDL has stayed a specialist's tool since the 1990s, useful mainly to people who already know its syntax. If a prompted LLM can reliably do that translation, QA teams could hook automated planners into bug triage without training testers to write formal goals themselves.
None of the six models cleared the ambiguity problem entirely, which is the part natural-language translation keeps rediscovering it can't just prompt its way past.