A new benchmark called oMeBench tests whether AI models actually understand organic chemistry or just pattern-match it. The answer, so far, is mostly the latter.
Researchers built oMeBench, a large-scale benchmark with over 10,000 expert-annotated mechanistic steps covering reaction types, intermediate structures, and difficulty ratings. They paired it with oMeS, a scoring system that checks both whether each reasoning step is logically consistent and whether the chemical structures it produces are actually valid. When they ran current large language models through it, the models showed decent chemical intuition but broke down across multi-step mechanisms, failing to keep intermediates consistent from one step to the next.
This matters because "AI can do chemistry" claims have mostly rested on models designing plausible-looking synthesis routes, not on proving they understand why those routes work. A model that gets the final answer right by guessing is not the same as one that can trace a mechanism step by step, and that difference matters if anyone plans to trust these systems with real reaction design or safety-critical predictions.
One wrinkle: the researchers found that fine-tuning smaller, open models with the right prompting let them match closed-source frontier models on this task, suggesting the bottleneck here is training data and technique, not raw scale.