- CPU‑only models can handle the bulk of LLM safety checks, researchers report.
The authors benchmarked five classic CPU classifiers—including support vector machines and gradient‑boosted trees using TF‑IDF features—against a small SSM model (Mamba‑130M) and two transformer GPU setups (DeBERTa‑v3 and Gemma‑2B with LoRA). Tests covered nine jailbreak sources across three regimes: in‑distribution (D1), out‑of‑distribution (D2), and adversarially obfuscated (D3). On D1, the top CPU model hit the same accuracy as the best GPU transformer while costing about 20% of the GPU deployment. In D2 the CPU models missed many attacks due to over‑confident false negatives, but on D3 they outperformed the GPU baselines by over 26 percentage points in F1 score. The paper bundles the findings into GuardChain, a three‑stage pipeline (regex → CPU → GPU) that routes each prompt to the cheapest stage that can decide confidently.
The result matters because most production traffic is benign; offloading 80 % of prompts to a cheap CPU stage cuts latency and hardware spend dramatically. It also shows that relying exclusively on GPU guards leaves a blind spot for out‑of‑distribution attacks.
In practice, operators can now consider hybrid pipelines rather than defaulting to expensive GPU inference for every safety check.