@inproceedings{joshi-etal-2025-adapting,
    title = "Adapting Multilingual {LLM}s to Low-Resource Languages using Continued Pre-training and Synthetic Corpus: A Case Study for {H}indi {LLM}s",
    author = "Joshi, Raviraj  and
      Singla, Kanishk  and
      Kamath, Anusha  and
      Kalani, Raunak  and
      Paul, Rakesh  and
      Vaidya, Utkarsh  and
      Chauhan, Sanjay Singh  and
      Wartikar, Niranjan  and
      Long, Eileen",
    editor = "Weerasinghe, Ruvan  and
      Anuradha, Isuri  and
      Sumanathilaka, Deshan",
    booktitle = "Proceedings of the First Workshop on Natural Language Processing for Indo-Aryan and Dravidian Languages",
    month = jan,
    year = "2025",
    address = "Abu Dhabi",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.indonlp-1.6/",
    pages = "50--57",
    abstract = "Multilingual LLMs support a variety of languages; however, their performance is suboptimal for low-resource languages. In this work, we emphasize the importance of continued pre-training of multilingual LLMs and the use of translation-based synthetic pre-training corpora for improving LLMs in low-resource languages. We conduct our study in the context of the low-resource Indic language Hindi. We introduce Nemotron-Mini-Hindi 4B, a bilingual SLM supporting both Hindi and English, based on Nemotron-Mini 4B. The model is trained using a mix of real and synthetic Hindi + English tokens, with continuous pre-training performed on 400B tokens. We demonstrate that both the base and instruct models achieve state-of-the-art results on Hindi benchmarks while remaining competitive on English tasks. Additionally, we observe that the continued pre-training approach enhances the model{'}s overall factual accuracy."
}