A new open-source benchmark called SAKE reveals that strong overall accuracy can mask real gaps in how well large language models handle software architecture.
Researchers introduced SAKE — Software Architectural Knowledge Evaluation — a benchmark of 2,154 expert-curated multiple-choice questions spread across eight architectural categories and four context-length levels. They tested 11 proprietary and open-weight models in both zero-shot and five-shot settings. The headline numbers look decent, but performance varied sharply by category, with meaningful gaps in areas that matter most to working architects: quality attribute trade-offs, design patterns, and system-level constraints.
Most coding benchmarks measure whether a model can write a function or spot a syntax error — tasks that have little to do with deciding between a monolith and a service mesh, or knowing when eventual consistency is an acceptable trade-off. SAKE targets that higher-order reasoning, giving evaluators a reproducible way to track whether models are actually getting better at it or just getting better at appearing to.
The benchmark, evaluation scripts, and full results are released publicly, which is the right move — proprietary leaderboards have a way of flattering the models that paid for them.