An LLM's screening decisions in systematic reviews can shift based on how you batch its queries, not just what data you show it.
Researchers tested large language models on binary classification tasks across five systematic reviews, the process of screening studies for inclusion or exclusion. They compared processing items one at a time against grouping them into batches, and tested runs with and without metadata about how common the target class was in each dataset. The prevalence metadata made little difference either way. Batch processing did the opposite: it produced larger shifts in model behavior, and those shifts varied depending on how skewed the class balance was. Aggregate accuracy numbers and item-level decisions did not always tell the same story.
That gap matters for anyone treating batching as a free cost optimization. Systematic reviews already run on tight budgets, and batching queries is an obvious way to cut API bills when screening thousands of abstracts. This study suggests that shortcut can quietly change which studies get included or excluded, even when overall performance metrics look stable.
It is a reminder that squeezing costs out of an LLM pipeline is never purely an engineering decision. If your evaluation only checks the top-line score, you can ship a batching change that looks free and isn't.