HI-CMLM: Improve CMLM with Hybrid Decoder Input
Minghan Wang, Guo Jiaxin, Yuxia Wang, Yimeng Chen, Su Chang, Daimeng Wei, Min Zhang, Shimin Tao, Hao Yang
Abstract
Mask-predict CMLM (Ghazvininejad et al.,2019) has achieved stunning performance among non-autoregressive NMT models, but we find that the mechanism of predicting all of the target words only depending on the hidden state of [MASK] is not effective and efficient in initial iterations of refinement, resulting in ungrammatical repetitions and slow convergence. In this work, we mitigate this problem by combining copied source with embeddings of [MASK] in decoder. Notably. it’s not a straightforward copying that is shown to be useless, but a novel heuristic hybrid strategy — fence-mask. Experimental results show that it gains consistent boosts on both WMT14 En<->De and WMT16 En<->Ro corpus by 0.5 BLEU on average, and 1 BLEU for less-informative short sentences. This reveals that incorporating additional information by proper strategies is beneficial to improve CMLM, particularly translation quality of short texts and speeding up early-stage convergence.- Anthology ID:
- 2021.inlg-1.16
- Volume:
- Proceedings of the 14th International Conference on Natural Language Generation
- Month:
- August
- Year:
- 2021
- Address:
- Aberdeen, Scotland, UK
- Editors:
- Anya Belz, Angela Fan, Ehud Reiter, Yaji Sripada
- Venue:
- INLG
- SIG:
- SIGGEN
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 167–171
- Language:
- URL:
- https://aclanthology.org/2021.inlg-1.16
- DOI:
- 10.18653/v1/2021.inlg-1.16
- Cite (ACL):
- Minghan Wang, Guo Jiaxin, Yuxia Wang, Yimeng Chen, Su Chang, Daimeng Wei, Min Zhang, Shimin Tao, and Hao Yang. 2021. HI-CMLM: Improve CMLM with Hybrid Decoder Input. In Proceedings of the 14th International Conference on Natural Language Generation, pages 167–171, Aberdeen, Scotland, UK. Association for Computational Linguistics.
- Cite (Informal):
- HI-CMLM: Improve CMLM with Hybrid Decoder Input (Wang et al., INLG 2021)
- PDF:
- https://preview.aclanthology.org/add_acl24_videos/2021.inlg-1.16.pdf