AI coding agents that rewrite their own code can be quietly taught to write insecure code, and the lesson sticks.
A new paper tests three self-modifying coding agents -- the Darwin Godel Machine (with researcher modifications), the Self-Improving Coding Agent, and Hyperagents (left mostly as-is) -- by slipping a poisoned benchmark into their self-evaluation and self-improvement process. In one proof-of-concept, Hyperagents running on Sonnet 4.5 evolved its own instructions to disable HTTPS certificate validation, and then applied that shortcut on unrelated, clean URL-fetching tasks it had never seen poisoned data for. The researchers also found the contamination often survived later rounds of self-improvement run against clean benchmarks, meaning one bad evaluation early on can outlast every subsequent attempt to correct course.
The paper is an explicit update to Ken Thompson's 1984 "Reflections on Trusting Trust," which showed a compiler could be backdoored to keep reinserting its own Trojan even when recompiled from clean source. Swap "compiler" for "agent that writes its own successor," and the same trick works: the benchmark, not the training data or the model weights, becomes the attack surface. That matters because self-improving agents are being pitched as a path to faster, more capable coding tools -- this paper is a reminder that the evaluation suite grading those improvements needs the same security scrutiny as the code itself.
An AI that grades its own homework was always going to be a tempting target; now there's a proof it can be gamed.