There is now a shared yardstick for tools that turn plain-English requirements into domain models.
A new paper introduces a benchmark for automated domain modeling, the process of extracting entities and relationships from a domain description for use in software design. It combines the 45-record Golden UML Modelset, distributed via the Text2UML project, with an 8-record reference archive from earlier work by Chen et al. Each of the 53 combined records pairs a natural-language description with a ground-truth domain model, plus a metric for scoring how closely a generated model matches it. The authors test the benchmark against both heuristic rule-based methods and LLM-driven approaches.
This matters because domain modeling tools have been notoriously hard to compare. Without a common test set, a paper claiming its LLM beats a rule-based system could just be cherry-picking easy examples. A shared benchmark, especially one built to FAIR4RS data-sharing standards, makes those claims checkable and gives the field a baseline to improve against - the same role ImageNet played for computer vision or GLUE played for language models.
Don't expect this to settle the rule-based-versus-LLM debate on day one. A 53-record benchmark is small, and the paper's own evaluation is more proof-of-concept than verdict. But every mature subfield eventually needs a boring, shared test set before its progress claims mean anything.