Stuffing a massive data schema into an LLM prompt is expensive, slow, and often counterproductive — a new paper proposes a smarter way to trim it down first.
Researchers introduced SchemaRAG, a framework that uses retrieval-augmented generation to dynamically prune large schemas before they reach the model. Instead of feeding an LLM a full blueprint of every possible output field, SchemaRAG uses schema metadata and few-shot examples to identify which parts of the schema are actually relevant to a given input. The team tested it on healthcare and e-commerce datasets — two domains notorious for sprawling, deeply nested data structures. Results showed up to an 8.8% improvement in micro-F1 scores, a 47% drop in latency, and a 48% reduction in token costs.
The cost and latency numbers matter more than the accuracy bump. Anyone running structured extraction at scale knows that ballooning prompts are a silent tax — and the "lost-in-the-middle" problem, where models ignore content buried deep in a long context, compounds the damage. SchemaRAG attacks both problems at once by keeping prompts lean without requiring a smaller or faster model.
The approach is a targeted answer to a pain point that grows more acute as enterprises push LLMs into complex data pipelines. RAG is already a standard tool for document retrieval; applying the same logic to schema retrieval is a logical extension — one that, on this evidence, pays off without the need for model fine-tuning or architectural overhaul.