A research team has released A2utoLPBench, a benchmark that generates its own linear programming problems instead of relying on a static hand-labeled dataset.
The system works by picking a feasible point and its dual variable first, then constructing a problem for which that point is provably optimal — meaning the correct answer is known before any solver is called and no human annotator is needed. The benchmark ships with a Docker image whose instructions are written for an LLM-driven agent to read directly, plus a reference solver-critic baseline. Any agent can run the full evaluation with a single command and get a calibrated score.
The contamination problem is real: once a benchmark dataset is public, its problems can end up in the training data of the very models being tested, turning evaluation into a memory quiz. A generator-based approach sidesteps that by producing fresh problems after any model's training cutoff. Difficulty is also tunable via two parameters — problem size n and constraint count m — which lets researchers stress-test agents at whatever level they need.
Fixed benchmarks have a shelf life measured in months once capable models appear; the interesting question is whether the research community adopts generator-based evaluation broadly, or treats this as a niche tool for LP specifically.