@inproceedings{bassani-etal-2021-clustering,
    title = "Clustering Monolingual Vocabularies to Improve Cross-Lingual Generalization",
    author = "Bassani, Riccardo  and
      S{\o}gaard, Anders  and
      Deoskar, Tejaswini",
    editor = "Ataman, Duygu  and
      Birch, Alexandra  and
      Conneau, Alexis  and
      Firat, Orhan  and
      Ruder, Sebastian  and
      Sahin, Gozde Gul",
    booktitle = "Proceedings of the 1st Workshop on Multilingual Representation Learning",
    month = nov,
    year = "2021",
    address = "Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Author-page-Marten-During-lu/2021.mrl-1.3/",
    doi = "10.18653/v1/2021.mrl-1.3",
    pages = "32--40",
    abstract = "Multilingual language models exhibit better performance for some languages than for others (Singh et al., 2019), and many languages do not seem to benefit from multilingual sharing at all, presumably as a result of poor multilingual segmentation (Pyysal o et al., 2020). This work explores the idea of learning multilingual language models based on clustering of monolingual segments. We show significant improvements over standard multilingual segmentation and training across nine languages on a question answering task, both in a small model regime and for a model of the size of BERT-base."
}