Researchers have released DiscoGen, a procedural task generator designed to help AI systems discover new machine learning algorithms on their own.
The system works by producing billions of algorithm discovery tasks across a range of machine learning subfields — think designing optimizers for reinforcement learning or loss functions for image classification. Tasks are parameterized so difficulty and complexity can be tuned. Alongside the generator, the team ships DiscoBench, a fixed subset of those tasks meant to give researchers a stable, apples-to-apples evaluation suite. The code is available on GitHub under an open-source license.
The motivation is less flashy than it sounds, but more important: existing benchmarks for so-called algorithm discovery agents are contaminated, saturated, or methodologically sloppy. If you are trying to build AI that invents better AI, a broken measuring stick is a serious problem. DiscoGen borrows the procedural generation idea from reinforcement learning research, where endlessly varied environments have long been used to stress-test agents without letting them memorize the course.
Whether billions of synthetic tasks translate into genuine algorithmic breakthroughs remains unproven — the paper demonstrates use mainly through automated prompt tuning experiments, which is a fairly modest starting point for a tool pitched at unlocking "new breakthroughs."