Researchers just built a report card for AI models on mechanical engineering, and most of them are still failing the hard questions.
The benchmark, called MecEng, gives large language models 84 tasks split across three difficulty levels, asking them to turn plain-text descriptions into working multibody simulation models. Easy tasks cover rigid bodies with joints and contact. Hard tasks require generating exact 3D geometry, running tetrahedral finite-element meshing, and applying Hurty-Craig-Bampton model reduction to flexible parts. The pipeline checks the AI-generated models against expert answers on multiple fronts, including whether the system graph matches, whether the numbers solve correctly, and whether mass, geometry, and eigenfrequencies line up. Across 32 open-weight models and two proprietary ones, the best open-weight model hit an 86.0% success rate on rigid-body tasks, versus 91.4% for the top proprietary model, while flexible multibody tasks proved considerably harder for everyone.
This is one of the first systematic attempts to test whether LLMs actually grasp physical space and mechanics, rather than just pattern-matching code syntax or algebra, which is where most existing benchmarks stop. The gap between near-90% scores on rigid-body setups and much rougher performance on flexible-body geometry suggests these models can look competent on tidy problems while still missing the deeper spatial reasoning that real engineering work demands.
A model that writes simulation code which compiles is not the same as one that understands the physical object it is supposedly modeling.