Researchers tried to find out whether large language models can rediscover classic computer science algorithms without just quoting them from memory.
A team used LLM unlearning techniques to erase a model's direct recall of a target algorithm, then asked it to reconstruct the method using only general reasoning. Unlearning doesn't scrub a model completely clean, so the researchers also ran a simpler test: blocking the algorithm's name from appearing during decoding. That alone dropped recovery rates to between 19% and 39%. Even with the heavier unlearning approach, models still failed to recover nearly half the target algorithms, and the ones they could reconstruct tended to be structurally simple.
The gap matters because AI labs increasingly pitch LLMs as partners in scientific discovery, not just assistants that summarize papers. If models stumble this badly on well-documented, decades-old algorithms, that's a rough signal for claims about genuinely novel invention. The researchers also built a generative verifier to stop models from devolving into incoherent reasoning loops - a problem serious enough that it needed its own fix.
Call it a reality check: the paper treats the unlearned model's performance as a generous upper bound on what today's LLMs can invent from scratch, and even that bound is unimpressive. The code is on GitHub, so anyone skeptical of the marketing can go check the math themselves.