@inproceedings{j-etal-2024-romansetu,
    title = "{R}oman{S}etu: Efficiently unlocking multilingual capabilities of Large Language Models via {R}omanization",
    author = "J, Jaavid  and
      Dabre, Raj  and
      M, Aswanth  and
      Gala, Jay  and
      Jayakumar, Thanmay  and
      Puduppully, Ratish  and
      Kunchukuttan, Anoop",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/add-emnlp-2024-awards/2024.acl-long.833/",
    doi = "10.18653/v1/2024.acl-long.833",
    pages = "15593--15615",
    abstract = "This study addresses the challenge of extending Large Language Models (LLMs) to non-English languages, specifically those using non-Roman scripts. We propose an approach that utilizes the romanized form of text as an interface for LLMs, hypothesizing that its frequent informal use and shared tokens with English enhance cross-lingual alignment. Our approach involve the continual pretraining of a English LLM like Llama 2 on romanized text of non-English, non-Roman script languages, followed by instruction tuning on romanized data. The results indicate that romanized text not only reduces token fertility by 2x-4x but also matches if not outperforms native script representation across various NLU, NLG and MT tasks. Moreover, the embeddings computed on romanized text exhibit closer alignment with their English translations than those from the native script. Our approach presents a promising direction for leveraging the power of English LLMs in languages traditionally underrepresented in NLP research."
}