A research team at the University of South Carolina has published details on Carolina Guide, an AI assistant designed to answer academic policy questions without crossing into territory that could get students into real trouble.
The system uses a multi-agent retrieval-augmented generation pipeline, meaning it pulls answers directly from official policy documents rather than generating them from model memory. Researchers tested it on 90 queries across six departments, hitting 98.9% retrieval success and a near-perfect mean reciprocal rank score. More meaningfully, they also threw 30 adversarial queries at the system's guardrail layer - requests for course recommendations or personalized guidance - and it correctly refused 86% of them while still answering 93% of legitimate questions.
Most university chatbot deployments skip the hard part: deciding what the system should never do. Carolina Guide's refusal architecture matters because the failure mode for policy chatbots isn't just wrong answers - it's confident wrong answers that students act on. Blocking personalized advising keeps liability with trained humans, where it belongs.
The paper is an implicit critique of off-the-shelf LLM deployments in institutional settings. Retrieval success metrics near 99% sound impressive, but the 90-query test set is small enough that statistical confidence is limited - the authors acknowledge this. Still, the architecture itself is the contribution: guardrails, citation enforcement, and departmental modularity over raw conversational fluency.