@inproceedings{lu-etal-2026-llms,
    title = "{LLM}s for Now, Fine-Tuning for Later: An Ensemble Approach to Data Drift in Domain-Specific Tasks",
    author = "Lu, Yuxuan  and
      Yao, Bingsheng  and
      Zhang, Shao  and
      Sang, Yisi  and
      Wang, Yun  and
      Gu, Hansu  and
      Zhang, Peng  and
      Lu, Tun  and
      Li, Toby Jia-Jun  and
      Wang, Dakuo",
    editor = "T.Y.S.S., Santosh  and
      Rodriguez, Juan Diego  and
      de Gibert, Ona",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics ({ACL} 2026)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-srw.77/",
    pages = "861--876",
    ISBN = "979-8-89176-393-7",
    abstract = "Deploying machine learning models in real-world domain-specific scenarios is challenged by the scarcity of expert annotations and by data drift, where the statistical properties of incoming data continuously evolve. Active Learning (AL) iteratively improves compact models with expert annotations but suffers from recurring cold-start degradation, while LLMs provide strong off-the-shelf performance yet cannot leverage newly accumulated labels, raising the question: how can we better leverage LLMs to assist the active learning process? Through an empirical study on five legal and biomedical datasets, we reveal a complementary temporal dynamic: LLMs excel during early and post-drift stages, while AL-assisted compact models eventually surpass them as annotations accumulate. Motivated by this finding, we propose an ensemble system that combines an LLM, an AL-assisted compact model, and an automatic switch module that routes predictions to the better-performing model in real time. Evaluated under simulated data drift on two mental health datasets, our system achieves 96{--}98{\%} switch accuracy and consistently outperforms either model used alone."
}