A subtle code flaw in agent-based policy models can cause optimizers to pick the wrong policy winner more than 80% of the time — and nobody notices because every individual stage looks correct.
Researchers audited EpidemiOptim, a peer-reviewed epidemic policy toolbox published in JAIR, and found three structurally independent ways the codebase extracts the same outcome metric from simulation data. They call the problem metric aggregation divergence, or MAD. In 500 independent runs, disagreement between pipeline stages on which policy "won" occurred in 64.2% of cases. A follow-up experiment using 300 seeds found the optimizer recommended the wrong champion in 83% of replications, with a mean welfare gap of 2.19 units and a Gini inequality gap of 0.050 units. Three of 249 flipped seeds also crossed statistical significance thresholds — meaning published conclusions could change.
What makes MAD tricky is that it is invisible at the stage level. Each chunk of code is internally consistent; the inconsistency only surfaces when you compare outputs across stages. That is a harder class of bug to catch than a straightforward coding error, and it is exactly the kind of thing peer review tends to miss. The Lake Problem DPS workflow, a separate public benchmark, illustrated the stakes: the archived published-path recommendation hit joint-threshold success at 0.401, while a unified contract path reached 0.552.
The proposed fix is a "metric contract" — a single shared callable enforced at dispatch time across all pipeline stages — which the authors say eliminates divergence by construction at roughly 3% runtime overhead. It is standard software engineering discipline applied to a research context that has not demanded it yet, which raises a fair question about how many other published agent-based policy results are quietly running the same bug.