Integrated Training for Sequence-to-Sequence Models Using Non-Autoregressive Transformer
Evgeniia Tokarchuk, Jan Rosendahl, Weiyue Wang, Pavel Petrushkov, Tomer Lancewicki, Shahram Khadivi, Hermann Ney
Abstract
Complex natural language applications such as speech translation or pivot translation traditionally rely on cascaded models. However,cascaded models are known to be prone to error propagation and model discrepancy problems. Furthermore, there is no possibility of using end-to-end training data in conventional cascaded systems, meaning that the training data most suited for the task cannot be used. Previous studies suggested several approaches for integrated end-to-end training to overcome those problems, however they mostly rely on(synthetic or natural) three-way data. We propose a cascaded model based on the non-autoregressive Transformer that enables end-to-end training without the need for an explicit intermediate representation. This new architecture (i) avoids unnecessary early decisions that can cause errors which are then propagated throughout the cascaded models and (ii) utilizes the end-to-end training data directly. We conduct an evaluation on two pivot-based machine translation tasks, namely French→German and German→Czech. Our experimental results show that the proposed architecture yields an improvement of more than 2 BLEU for French→German over the cascaded baseline.- Anthology ID:
- 2021.iwslt-1.32
- Volume:
- Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021)
- Month:
- August
- Year:
- 2021
- Address:
- Bangkok, Thailand (online)
- Editors:
- Marcello Federico, Alex Waibel, Marta R. Costa-jussà, Jan Niehues, Sebastian Stuker, Elizabeth Salesky
- Venue:
- IWSLT
- SIG:
- SIGSLT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 276–286
- Language:
- URL:
- https://aclanthology.org/2021.iwslt-1.32
- DOI:
- 10.18653/v1/2021.iwslt-1.32
- Cite (ACL):
- Evgeniia Tokarchuk, Jan Rosendahl, Weiyue Wang, Pavel Petrushkov, Tomer Lancewicki, Shahram Khadivi, and Hermann Ney. 2021. Integrated Training for Sequence-to-Sequence Models Using Non-Autoregressive Transformer. In Proceedings of the 18th International Conference on Spoken Language Translation (IWSLT 2021), pages 276–286, Bangkok, Thailand (online). Association for Computational Linguistics.
- Cite (Informal):
- Integrated Training for Sequence-to-Sequence Models Using Non-Autoregressive Transformer (Tokarchuk et al., IWSLT 2021)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2021.iwslt-1.32.pdf