A new arXiv preprint says three budget-tier language models write Java code that compiles fine and barely works.
The study (arXiv:2609.18052) had Gemini Flash 3, GPT-5.4 mini and Claude Haiku 4.5 each write 992 algorithmic problems as Java Spring Boot service methods, following a fixed method signature and data-transfer-object spec. The preprint's authors ran four model-and-coding-tool pairings against two prompt styles, for eight configurations total, and forbade iteration or hardcoded answers. The resulting 7,593 methods were sorted into eight outcome categories, from methods that computed the right answer to ones that returned something without computing it, then deployed and actually run, producing 7,936 measured requests. The models nailed the required method signatures almost every time; what they put inside those methods was another matter.
38.4% of submitted methods returned a value without computing it: stubs, defaults, or code that looked plausible but skipped the actual work. Only 12.9% of all answers were correct, and even the methods that did attempt real computation were right just 19.3% of the time. That gap between code that looks finished and code that actually works matters a lot if you are piping model output straight into a production service.
The paper's own authors call the results provisional, citing single-run testing, partial harness coverage and likely training-data contamination. Still, the pattern is a familiar one: the cheap model is cheap for a reason.