@inproceedings{ratnakar-etal-2026-latentgate,
    title = "{L}atent{G}ate: Low-Latency Semantic Routing via Frozen-Backbone Probing of Small Language Models",
    author = "Ratnakar, Shivam  and
      Talasila, Abhiroop  and
      Doifode, Vinayak K",
    editor = "Li, Yunyao  and
      Rehm, Georg  and
      Tu, Mei",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 6: Industry Track)",
    month = jul,
    year = "2026",
    address = "San Diego, California, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-form-platform/2026.acl-industry.153/",
    pages = "2284--2294",
    ISBN = "979-8-89176-394-4",
    abstract = "As Multi-Agent Systems scale to hundreds of specialized agents, routing becomes a critical bottleneck. Prompt-based LLM routers deliver strong semantic reasoning but incur prohibitive latency ({\textasciitilde}1500{--}2000ms) and cost that scales with agent count, while embedding-based routers are fast (25{--}50ms) but collapse semantically similar yet functionally distinct agents. We identify *representation anisotropy*, the geometric collapse of hidden-state vectors into a narrow cone, as a key mechanism underlying embedding-based routing failure. We propose **LatentGate**, a non-generative router that extracts mean-pooled hidden states from a frozen small language model (SLM), applies PCA-whitening to resolve the anisotropy, and trains a lightweight linear probe for agent classification. Across 5 SLM backbones and 100 enterprise agents, LatentGate achieves 98.8{\%} in-domain and 80.0{\%} OOD accuracy on natural queries, 13{--}22 absolute points above embedding baselines, and 92.9{\%} on CLINC150. It takes {\textasciitilde}28ms to run on a T4 GPU, with the SLM forward pass independent of agent count and classification adding a negligible O(Ck) term. We demonstrate the potential of using a lightweight linear probe to enable sub-10ms warm-start retraining from user feedback, providing a foundation for continual learning in production environments. Benchmarking prompt-based routing with GPT-4.1, GPT-4.1-nano, and Gemini 2.5 Flash confirms degradation to 70{--}77{\%} accuracy at 100 agents with 1500{--}2000ms latency, motivating non-generative alternatives."
}