A new paper shows the statistical safety net under AI-powered intrusion detection tears the moment an attacker leans on it.
Researchers tested large language models fine-tuned to spot network intrusions, wrapped in a technique called conformal prediction, which is supposed to mathematically guarantee that predictions are right a known percentage of the time. That guarantee held on clean traffic but fell apart across three intrusion-detection benchmarks once an attacker tweaked the network features the model relies on. A first fix, recalibrating on the kind of manipulated traffic an attacker is expected to send, restored the guarantee, but only if you know the attacker's method in advance. A sharper attacker who probes the model's own score could still break it.
That gap matters because conformal prediction is the selling point of these systems: the promise that a security team can trust the confidence numbers, not just the labels. The researchers' actual fix is more mundane than clever math - strip out any feature an attacker can control, plus everything downstream of it, before scoring. That turns a probabilistic promise into an absolute one, but it costs seven to fourteen points of clean accuracy across three different model architectures.
In other words, the more attacker-proof the guarantee, the less the detector actually sees - a trade-off any security buyer should ask about before trusting the confidence score on the dashboard.