AI agents built to predict research trends are getting outperformed by a basic moving average.
Researchers introduced Research Attention Prediction (RAP), a benchmark covering 278 AI/ML fields across 1,390 episodes. At each test point, an LLM agent searches a time-restricted slice of arXiv and predicts how paper shares across eight frozen research directions will shift over the next six months. The study separates two different measures: compositional accuracy, which scores the full predicted distribution, and future-specific updating, which isolates how well a model reacts to genuinely new information once historical activity is already known. On compositional accuracy, all four models tested performed worse than a simple exponentially weighted moving average (EWMA) baseline, even though enabling web search generally helped each model. On future-specific updating, only GPT-5.5 with search re-enabled slightly beat that same EWMA baseline.
The gap matters because it undercuts a common assumption about AI research agents: that search access plus a language model should beat dumb statistical extrapolation. Instead, the paper traces part of the failure to a design choice: models that carry forward a running state do better than models asked to forecast directly from scratch, partly because forecast-first approaches pull in less recent evidence during their searches.
There is a silver lining for anyone building these systems: fine-tuning Qwen3-4B on real outcomes lifted its forecast Spearman correlation by 0.105 on held-out fields it had not seen during training. Grand claims about AI-run research pipelines should probably wait for that kind of tuning to become standard, not the exception.