@inproceedings{ebing-etal-2026-one,
    title = "One Script Instead of Hundreds? On Pretraining {R}omanized Encoder Language Models",
    author = "Ebing, Benedikt  and
      Keller, Lennart  and
      Glava{\textbackslash}v{\{}s{\}}, Goran",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1909/",
    pages = "38291--38307",
    ISBN = "979-8-89176-395-1",
    abstract = "Exposing latent lexical overlap, script romanization has emerged as an effective strategy for improving cross-lingual transfer (XLT) in multilingual language models (mLMs). Most prior work, however, focused on setups that favor romanization the most: **(1)** transfer from high-resource Latin-script to low-resource non-Latin-script languages and/or **(2)** between genealogically closely related languages with different scripts. It thus remains unclear whether romanization is a good representation choice for *pretraining* general-purpose mLMs, or, more precisely, if information loss associated with romanization harms performance for high-resource languages. We address this gap by pretraining encoder LMs from scratch on both romanized and original texts for six typologically diverse high-resource languages, investigating two potential sources of degradation: **(i)** loss of script-specific information and **(ii)** dilution of language-specific representations from increased subword overlap. Using two romanizers with different fidelity profiles, we observe negligible performance loss for languages with segmental scripts, whereas languages with morphosyllabic scripts (Chinese and Japanese) suffer degradation that higher-fidelity romanization mitigates but cannot fully recover. Importantly, comparing monolingual LMs with their mLM counterpart, we find no evidence that increased subword overlap dilutes language-specific representations. We further show that romanization improves encoding efficiency (i.e., fertility) for segmental scripts at a negligible performance cost."
}