@inproceedings{casas-etal-2020-combining,
    title = "Combining Subword Representations into Word-level Representations in the Transformer Architecture",
    author = "Casas, Noe  and
      Costa-juss{\`a}, Marta R.  and
      Fonollosa, Jos{\'e} A. R.",
    editor = "Rijhwani, Shruti  and
      Liu, Jiangming  and
      Wang, Yizhong  and
      Dror, Rotem",
    booktitle = "Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/add-emnlp-2024-awards/2020.acl-srw.10/",
    doi = "10.18653/v1/2020.acl-srw.10",
    pages = "66--71",
    abstract = "In Neural Machine Translation, using word-level tokens leads to degradation in translation quality. The dominant approaches use subword-level tokens, but this increases the length of the sequences and makes it difficult to profit from word-level information such as POS tags or semantic dependencies. We propose a modification to the Transformer model to combine subword-level representations into word-level ones in the first layers of the encoder, reducing the effective length of the sequences in the following layers and providing a natural point to incorporate extra word-level information. Our experiments show that this approach maintains the translation quality with respect to the normal Transformer model when no extra word-level information is injected and that it is superior to the currently dominant method for incorporating word-level source language information to models based on subword-level vocabularies."
}