@inproceedings{zhu-etal-2025-multi,
    title = "Multi-Task Pre-Finetuning of Lightweight Transformer Encoders for Text Classification and {NER}",
    author = "Zhu, Junyi  and
      Ozkan, Savas  and
      Maracani, Andrea  and
      Mutlu, Sinan  and
      Min, Cho Jung  and
      Ozay, Mete",
    editor = "Potdar, Saloni  and
      Rojas-Barahona, Lina  and
      Montella, Sebastien",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: Industry Track",
    month = nov,
    year = "2025",
    address = "Suzhou (China)",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-industry.118/",
    pages = "1674--1686",
    ISBN = "979-8-89176-333-3",
    abstract = {Deploying natural language processing (NLP) models on mobile platforms requires models that can adapt across diverse applications while remaining efficient in memory and computation. We investigate pre-finetuning strategies to enhance the adaptability of lightweight BERT-like encoders for two fundamental NLP task families: named entity recognition (NER) and text classification. While pre-finetuning improves downstream performance for each task family individually, we find that na{\"i}ve multi-task pre-finetuning introduces conflicting optimization signals that degrade overall performance. To address this, we propose a simple yet effective multi-task pre-finetuning framework based on task-primary LoRA modules, which enables a single shared encoder backbone with modular adapters. Our approach achieves performance comparable to individual pre-finetuning while meeting practical deployment constraint. Experiments on 21 downstream tasks show average improvements of +0.8{\%} for NER and +8.8{\%} for text classification, demonstrating the effectiveness of our method for versatile mobile NLP applications.}
}