A fresh benchmark finds that large language models still can't reliably pull threat indicators out of malicious scripts without running them.
Researchers built SCRIPTIOC-BENCH, a set of 634 manually verified JavaScript, PowerShell, and VBScript malware samples, to test how well LLMs can statically extract indicators of compromise - URLs, domains, IP addresses, and filesystem artifacts - without executing the code. They graded models on whether IOCs were directly exposed in the script or buried behind encoding and obfuscation that required reconstruction. Across a range of proprietary and open-weight models, the best performer topped out at 65.4 F1, a measure that balances precision and recall. The team also built a taxonomy of false positives and tested two fixes on a small open-weight model - deterministic string utilities and task-specific fine-tuning - which improved precision and shifted remaining errors toward ones grounded in the actual sample.
Security teams increasingly lean on LLMs to triage script-based malware, one of the most common delivery methods for phishing payloads and living-off-the-land attacks. A 65.4 F1 ceiling means even the best model still misses or mislabels roughly a third of the actionable intelligence analysts need to block a domain or flag a malicious IP - not a rounding error when minutes matter during an incident. The findings also suggest static analysis alone isn't enough; the paper's own mitigations point toward pairing LLMs with deterministic tooling and targeted fine-tuning rather than expecting a general-purpose model to catch everything.
Treat any vendor pitch promising fully automated IOC extraction with the same skepticism you'd apply to a script claiming to be a PDF.