Security classifiers can be fooled in two ways at once — and most evaluations only catch one.
Researchers tested adversarial attacks against Random Forest and XGBoost models trained on four tabular security datasets covering phishing URLs, network intrusion, and IoT traffic. They evaluated five attack methods, including three that work as black-box attacks against tree-based models that have no gradients to exploit. The central finding: a gradient-based black-box method called ZOO produced a misleadingly high robustness score of around 0.98 against XGBoost — not because XGBoost was robust, but because ZOO was fooled by the model's piecewise-constant prediction surface. A score-based attack called Square Attack told a different story, exposing genuine vulnerability with a robustness score around 0.36.
The more unsettling result involves explanations. Security teams increasingly rely on SHAP values to understand why a model flagged something — that reasoning is what lets an analyst decide whether to escalate or dismiss an alert. The researchers introduce the Explainability Stability Index (ESI), a companion metric measuring how much SHAP attributions drift under adversarial pressure. Even when ZOO attacks appeared to fail against XGBoost, they still caused substantial explanation drift, with ESI scores between 0.06 and 0.16. Random Forest fared worse on that axis, with ESI between 0.14 and 0.29. Prediction robustness and explanation stability, the paper argues, are distinct and need to be measured together.
The practical implication is uncomfortable: a classifier that looks robust on standard metrics can still be feeding analysts corrupted reasoning — which is arguably the more dangerous failure mode in a triage workflow where humans act on explanations, not raw scores.