A new benchmark reveals that AI investment assistants can sound like expert investors without actually reasoning like one.
Researchers released InvestPhilBench v0.6, designed to test whether LLMs can reconstruct and apply the decision frameworks used by expert investors across eight cognitive tiers. The benchmark includes 118 verified principle cards, 25 decision-framework cards, and 243 questions spanning tasks from identifying basic principles (L1) to extrapolating novel frameworks (L8). An early four-model test found a wide provider-tier split, with composite scores ranging from 0.906 to 0.438. But the composite metric saturated quickly at the frontier, while a per-step measure called Gate Reconstruction Accuracy exposed a gap: the top-scoring model hit roughly 0.77 at L4 difficulty and fell to 0.57-0.62 at L7.
This is the benchmark research version of a problem that keeps appearing: holistic scoring rewards models for sounding right, not for being right. In investment research, where the specific sequence of a decision process can make a recommendation sound or reckless, that distinction is the whole point.
The researchers validated automated scoring against 100 expert-annotated examples at a Pearson correlation of 0.72; a cleaner, de-confounded leaderboard is promised for v1.0, meaning today's numbers stress-test the method more than they rank the models.