@inproceedings{gurgurov-etal-2025-multilingual-encoder,
    title = "On Multilingual Encoder Language Model Compression for Low-Resource Languages",
    author = "Gurgurov, Daniil  and
      Gregor, Michal  and
      Genabith, Josef Van  and
      Ostermann, Simon",
    editor = "T.y.s.s, Santosh  and
      Shimizu, Shuichiro  and
      Gong, Yifan",
    booktitle = "The 14th International Joint Conference on Natural Language Processing and The 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics",
    month = dec,
    year = "2025",
    address = "Mumbai, India",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-ijcnlp-aacl/2025.ijcnlp-srw.5/",
    pages = "47--58",
    ISBN = "979-8-89176-304-3",
    abstract = "In this paper, we combine two-step knowledge distillation, structured pruning, truncation, and vocabulary trimming for extremely compressing multilingual encoder-only language models for low-resource languages. Our novel approach systematically combines existing techniques and takes them to the extreme, reducing layer depth, feed-forward hidden size, and intermediate layer embedding size to create significantly smaller monolingual models while retaining essential language-specific knowledge. We achieve compression rates of up to 92{\%} while maintaining competitive performance, with average drops of 2{--}10{\%} for moderate compression and 8{--}13{\%} at maximum compression in four downstream tasks, including sentiment analysis, topic classification, named entity recognition, and part-of-speech tagging, across three low-resource languages. Notably, the performance degradation correlates with the amount of language-specific data in the teacher model, with larger datasets resulting in smaller performance losses. Additionally, we conduct ablation studies to identify the best practices for multilingual model compression using these techniques."
}