How you write an algorithm specification affects whether an LLM implements it correctly — but mostly when the model is small.
Researchers tested seven formats for describing machine learning algorithms — plain prose, LaTeX-style pseudocode, PDF-extracted pseudocode, Markdown fields, YAML-like specifications, JSON-like specifications, and Python code stubs — across five tasks, three models, and four experimental settings, producing 4,020 generated implementations. Hidden tests checked the specifics that most commonly break implementations: tie-breaking rules, array shapes, numerical rules, return structures, and invalid-input behavior. When only core information was provided, LaTeX-style pseudocode showed the largest average format effect, with YAML-like specifications and plain prose close behind. Code stubs, despite specifying function signatures up front, did not consistently improve correctness.
The results carry practical weight for anyone using LLMs to translate research papers into running code, a workflow that has grown common as model capabilities have expanded. Most papers leave implementation details implicit, which means a format choice can silently determine whether an LLM reproduces the right behavior or a plausible-looking approximation of it.
The study's format effects mostly disappeared under complete-information conditions for one of the three tested models — an OpenAI model identified in the paper by a version string not confirmed against any public release lineup, and omitted here on that basis — while the two smaller open models, Gemma 3 4B and Llama 3.2 3B, still showed format sensitivity even then. If format dependence shrinks as models scale up, the practical advice holds either way: state the interface, the numerical rules, and the edge cases explicitly rather than trusting any format to do that work.