The best large language models still can't match a seasoned aviation professional on domain-specific operational knowledge, according to a new benchmark called Pre-Flight.
Researchers released Pre-Flight, an open-source set of 300 multiple-choice questions drawn from ICAO and US FAA regulations, international airport ground operations standards, and complex operational scenarios. Questions were written and reviewed by practitioners with backgrounds in air traffic management, ground operations, and commercial flying. The benchmark runs on the Inspect evaluation framework and maintains a rolling leaderboard as new models ship. The strongest model tested — released in 2026 — scored 82.7%, up only modestly from roughly 75% in early 2025.
That gap matters because aviation is already seeing proposals to use LLMs for documentation, training generation, and customer-facing tools. A 12-plus percentage-point shortfall against an informal expert reference of around 95% is a meaningful reliability gap in a regulated, high-stakes domain — even for tasks the researchers describe as non-safety-critical operations. The slow pace of improvement, roughly 7-8 points over more than a year, suggests the ceiling isn't rising fast.
The benchmark is available inside the inspect_evals community package, which lowers the bar for others to test models as they release — a useful pressure valve given how quickly labs push new versions. Whether airlines and aviation vendors will wait for models to close that gap before deploying them is a separate question the benchmark can't answer.