A research proposal argues that compliance in regulated LLM deployments should be structural, not bolted on after the fact.
The paper, posted to arXiv, describes a system called ComplianceGate that inserts a trained encoder classifier in front of any LLM inference. Before a single token is decoded, the classifier evaluates each query for complexity and data sensitivity in about 7ms, then routes it to the right model in the right geographic location. Queries containing personally identifiable information go to a local endpoint, making it structurally impossible for that data to cross a jurisdictional boundary. Simpler queries hit smaller, cheaper models. The team tested the approach on 600 queries and reported a 39% median latency reduction, cost savings of 33-52% depending on query mix, and generation throughput of 122-200 tokens per second compared to 50-64 for the single-model baseline. The classifier hit 99.2% accuracy with near-perfect PII recall.
Most regulated-industry LLM deployments treat compliance as a filter applied after the fact, which means sensitive data can reach the wrong endpoint before anyone stops it. This architecture flips that: the routing decision happens pre-inference, so a data residency violation is not a policy failure to catch but a path that does not exist. That distinction matters a lot to healthcare, finance, and legal teams whose liability hinges on where data lands, not just what happens to it afterward.
The authors note that Mixture-of-Experts models do not solve this problem — routing between experts happens inside the model after data has already arrived at the endpoint. ComplianceGate is a pre-model layer, which is either an obvious idea whose time has come or a sign that the big foundation model providers have left a real gap in their enterprise offerings.