Smaller training runs can outperform bigger ones — at least in cybersecurity AI.
A team of researchers applied Domain-Adaptive Continuous Pretraining to three existing open models: Llama-3.1-8B, DeepSeek-R1-Distill-Qwen-14B, and Llama-3.3-70B-Instruct. They fed each a curated 126-million-word corpus drawn from security standards, academic papers, and technical documentation, running training across multi-node GPU clusters using a distributed sharding approach. The resulting models were then tested on three cybersecurity benchmarks — CTI-MCQ, CyberMetric, and SecEval — where all three improved after adaptation.
The headline result is efficiency. The best-performing adapted model, based on Llama-3.3-70B, hit accuracy scores of 0.718, 0.933, and 0.864 on those benchmarks, beating out specialized models like Llama-Primus-Base and Foundation-Sec-8B that were trained on 2.77 billion and 5 billion tokens respectively. The new approach used only 118.8 million tokens — a 23-to-42-fold reduction in training data. That gap matters because compute and energy costs scale with data volume, and purpose-built security models are expensive to maintain.
The results push back against the assumption that domain specialization requires purpose-built models trained from scratch on massive corpora. General-purpose models have eaten that argument before in other verticals; now security is getting the same treatment. Whether these adapted models hold up in production threat analysis — where edge cases and adversarial inputs are the norm — is the question the benchmarks do not answer.