Constrained decoding, the standard fix for LLMs that mangle JSON, has a hidden failure mode — and a new paper proposes a training-free workaround.
When developers need LLMs to return valid JSON, API calls, or other structured formats, they typically use constrained decoding: a technique that blocks any token that would break the syntax, token by token. The problem, according to researchers behind a new arXiv paper, is that this masking can steer a model away from what it was actually going to say — forcing it down locally valid but semantically wrong paths. Their proposed fix, Draft-Conditioned Constrained Decoding (DCCD), runs the model twice: first unconstrained to capture intent, then constrained but anchored to that initial draft. No fine-tuning required.
The gains are hard to ignore. On the GSM8K math reasoning benchmark, a 1-billion-parameter model jumped from 15.2% to 39.0% strict structured accuracy with DCCD — a 24-point swing. More pointedly, smaller model pairs using DCCD matched or beat much larger models running standard constrained decoding, which reframes the structured-output problem as one of inference strategy rather than raw scale.
That last finding lands at an awkward moment for the scaling-is-everything crowd. If a smarter two-pass decoding loop can close the gap between a 1B and a much larger model on structured tasks, the industry's reflex to throw more parameters at output reliability looks increasingly like an expensive habit.