Automated tools that patch old software against known vulnerabilities look a lot less impressive once you test them outside their comfort zone.
Researchers built Porting Benchmark, a dataset of 1,234 security patch backporting cases covering cross-version, cross-branch, and cross-repository scenarios, then ran five existing tools through a shared evaluation framework instead of each tool's own test setup. Under that common protocol, PortGPT and TSBPort held up reasonably well, but FixMorph and Mystique performance dropped substantially. The best tool's commit-level success rate fell from 85.2% on the simplest patch type to just 24.0% on the most structurally complex one. The team also traced failures to four causes: tools missing awareness of target APIs, semantic mismatches across versions, failures propagating non-local dependencies, and outright errors in locating or constructing the patch.
The gap matters because backporting security patches is exactly the kind of unglamorous, high-volume work maintainers hope AI can offload, keeping older software branches safe from N-day vulnerabilities already patched elsewhere. A tool that works great on the one repository it was built for is not proof it will work on the messy, varied reality of production codebases. The paper's dynamic-validation subset adds another wrinkle: exact-match scoring against a reference patch both under-credits legitimate but different fixes and misses cases that pass static comparison but fail to actually run.
Vendors touting 80%-plus success rates are, per this study, mostly reporting the score for patches, not the score for reality.