LLM prompt security research has a measurement problem, and a new academic paper is trying to fix it.
A team of researchers published a systematization-of-knowledge study cataloging the current state of jailbreak attacks, defenses, and evaluation methods for large language models. The paper introduces linked taxonomies that separate technical attack mechanisms from attacker and defender capabilities, and formalizes assumptions about model access, threat models, and cost budgets as explicit metadata. To back the framework, the authors released three artifacts: JailbreakDB (a curated dataset), PromptSecurity-Eval (a benchmark suite), and PromptSecurity, a modular platform that represents each experiment as a tuple of model, attack, defense, dataset, and judger.
The core finding is that current research is nearly impossible to compare. Attack success rates and claimed defense gains shift depending on which judger you use, whether you have white-box or black-box access to the model, how much compute you spend on the attack, and how the underlying model behaves on harmful queries before any attack is applied. That inconsistency means a defense that looks strong in one paper may be meaningless - or even counterproductive - under a different setup. A public leaderboard accompanies the release to let researchers run matched evaluations.
This is the kind of unglamorous infrastructure work that fields need before they can make real progress. LLM security has spent two years generating papers that don't talk to each other; a shared taxonomy and reproducible benchmark won't end jailbreaks, but they at least make it possible to know whether a claimed fix actually fixes anything.