Researchers find that training AI models to reverse-engineer Dart app binaries frequently backfires — and the metrics most labs use to measure success hide the damage.
A team published a systematic study testing six fine-tuned model variants across three base architectures ranging from 4B to 8B parameters on a new 154-task benchmark called HumanEval-Dart. The core finding is blunt: no fine-tuning configuration produced a statistically significant improvement on pass@k, the metric that measures whether decompiled code actually runs correctly. Fine-tuning the strongest base model, Qwen3-8B, caused a statistically significant regression of -5.65 percentage points. A separate experiment found that mixing in Swift training data hurt 4B models significantly but had no measurable effect on 8B models — a result the authors attribute to scaling effects.
The metric problem is arguably the more consequential finding. CodeBLEU and compile@k — two widely used proxies for code quality — improved in cases where pass@k moved in the opposite direction. That gap matters because most published work on neural decompilation leans on those cheaper-to-compute metrics, which means the field may be congratulating itself on gains that disappear under real-world conditions. The authors also flag assembly sequence length as the strongest predictor of task difficulty, with a capability cliff around 200 instructions.
Dart decompilation is a narrow slice of the reverse-engineering landscape, but the metric-validity problem the study exposes is not. Any lab fine-tuning code models and reporting CodeBLEU improvements should read this as a cautionary note: surface similarity is not correctness.