Abstract
The softmax layer in neural machine translation is designed to model the distribution over mutually exclusive tokens. Machine translation, however, is intrinsically uncertain: the same source sentence can have multiple semantically equivalent translations. Therefore, we propose to replace the softmax activation with a multi-label classification layer that can model ambiguity more effectively. We call our loss function Single-label Contrastive Objective for Non-Exclusive Sequences (SCONES). We show that the multi-label output layer can still be trained on single reference training data using the SCONES loss function. SCONES yields consistent BLEU score gains across six translation directions, particularly for medium-resource language pairs and small beam sizes. By using smaller beam sizes we can speed up inference by a factor of 3.9x and still match or improve the BLEU score obtained using softmax. Furthermore, we demonstrate that SCONES can be used to train NMT models that assign the highest probability to adequate translations, thus mitigating the “beam search curse”. Additional experiments on synthetic language pairs with varying levels of uncertainty suggest that the improvements from SCONES can be attributed to better handling of ambiguity.- Anthology ID:
- 2022.naacl-main.365
- Volume:
- Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
- Month:
- July
- Year:
- 2022
- Address:
- Seattle, United States
- Editors:
- Marine Carpuat, Marie-Catherine de Marneffe, Ivan Vladimir Meza Ruiz
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 4950–4961
- Language:
- URL:
- https://aclanthology.org/2022.naacl-main.365
- DOI:
- 10.18653/v1/2022.naacl-main.365
- Cite (ACL):
- Felix Stahlberg and Shankar Kumar. 2022. Jam or Cream First? Modeling Ambiguity in Neural Machine Translation with SCONES. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4950–4961, Seattle, United States. Association for Computational Linguistics.
- Cite (Informal):
- Jam or Cream First? Modeling Ambiguity in Neural Machine Translation with SCONES (Stahlberg & Kumar, NAACL 2022)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2022.naacl-main.365.pdf