A new academic benchmark tests LLM systems on German legal reasoning — and the results favor the big closed models.
BenGER (Benchmark for German Law) combines 596 exam-style case tasks drawn from multiple levels of legal education with 531 shorter doctrinal reasoning tasks, totaling 1,127 problems. Researchers evaluated 12 systems spanning closed flagship, efficiency-oriented, and open-weight categories. Grading used a rubric-aligned LLM-as-a-Judge method cross-validated against a three-blind-human-reviewer pool — six judge families total. The LLM judge correlated with human graders at Pearson r=0.76 and Cohen's kappa=0.60, and two judges from independent providers cleared the Calderon single-reviewer replacement bar, meaning they performed as reliably as a single qualified human reviewer on human-authored solutions.
Closed flagship models led across all three corpora, which will not surprise anyone who has watched these leaderboards. More interesting: human-AI co-creation measurably outperformed unaided human work on the same tasks, adding a data point to the ongoing debate about whether AI is a lawyer's tool or a lawyer's replacement.
German legal reasoning is a useful stress test because subsumption — applying a general rule to specific facts in structured steps — is explicit and gradeable in ways that common-law argumentation often is not. Whether the 12 unnamed systems here include the models your firm is already using is, inconveniently, left as an exercise for the reader.