@inproceedings{purason-etal-2024-mixing,
    title = "Mixing and Matching: Combining Independently Trained Translation Model Components",
    author = {Purason, Taido  and
      T{\"a}ttar, Andre  and
      Fishel, Mark},
    editor = {V{\'a}zquez, Ra{\'u}l  and
      Mickus, Timothee  and
      Tiedemann, J{\"o}rg  and
      Vuli{\'c}, Ivan  and
      {\"U}st{\"u}n, Ahmet},
    booktitle = "Proceedings of the 1st Workshop on Modular and Open Multilingual NLP (MOOMIN 2024)",
    month = mar,
    year = "2024",
    address = "St Julians, Malta",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/add-emnlp-2024-awards/2024.moomin-1.5/",
    pages = "44--56",
    abstract = "This paper investigates how to combine encoders and decoders of different independently trained NMT models. Combining encoders/decoders is not directly possible since the intermediate representations of any two independent NMT models are different and cannot be combined without modification. To address this, firstly, a dimension adapter is added if the encoder and decoder have different embedding dimensionalities, and secondly, representation adapter layers are added to align the encoder`s representations for the decoder to process. As a proof of concept, this paper looks at many-to-Estonian translation and combines a massively multilingual encoder (NLLB) and a high-quality language-specific decoder. The paper successfully demonstrates that the sentence representations of two independent NMT models can be made compatible without changing the pre-trained components while keeping translation quality from deteriorating. Results show improvements in both translation quality and speed for many-to-one translation over the baseline multilingual model."
}