A new benchmark called AeroCopilotBench puts AI copilots through simulated cockpit emergencies instead of just quizzing them on aviation trivia, and most models still stumble at actually doing the job.
Researchers built the AeroCopilot Operational Environment, a virtual cockpit that converts aircraft manufacturers' Pilot's Operating Handbook procedures into executable state transitions with hard safety constraints. AeroCopilotBench runs on top of it as a two-tier test: Tier 1 is 1,200 multiple-choice aviation knowledge questions, and Tier 2 drops models into 73 emergency and abnormal scenarios where they must diagnose the fault and operate cockpit systems through a standardized tool interface. A trajectory only counts as a success if the model hits every task goal without breaking a single safety rule, with goal progress and safety tracked separately. Across 12 models tested, the best Tier 2 success rate was 72.6 percent, and scoring well on the knowledge quiz did not reliably predict who could run the actual checklist.
Most AI aviation demos so far have been about answering questions, not doing the job, and this is one of the first attempts to separate sounding like a pilot from operating like one under real safety constraints. Digging into 451 failed runs from three models, the researchers found mostly mundane, procedural errors: agents skipped steps, ignored what the cockpit state was telling them, and lost track of long multi-step tasks - the kind of mistakes that ground student pilots, not exotic reasoning failures.
It's a useful reminder that acing a knowledge exam is not the same as being trusted to touch the throttle, a distinction worth keeping in mind as "AI agent" gets attached to every task-automation pitch this year.