Abstract
Multilingual understanding models (or encoder-based), pre-trained via masked language modeling, have achieved promising results on many language understanding tasks (e.g., mBERT). However, these models are not capable of generating high-quality text compared with decoder-based causal language models. Can we transform a pre-trained language understanding model into an effective language generation model? We propose a Semantic-Guided Alignment-then-Denoising (SGA) approach to adapt a multilingual encoder to a multilingual generator with a small number of additional parameters. Experiments show that the proposed approach is an effective adaption method, outperforming widely-used initialization-based methods with gains of 9.4 BLEU on machine translation, 8.1 Rouge-L on question generation, and 5.5 METEOR on story generation on XLM-Rlarge. On the other hand, we observe that XLM-R is still inferior to mBART in supervised settings despite better results on zero-shot settings, indicating that more exploration is required to make understanding models strong generators. Our code is available at https://github.com/chengzhipanpan/XLMR4MT.- Anthology ID:
- 2023.findings-emnlp.1031
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 15432–15444
- Language:
- URL:
- https://aclanthology.org/2023.findings-emnlp.1031
- DOI:
- 10.18653/v1/2023.findings-emnlp.1031
- Cite (ACL):
- Bohong Wu, Fei Yuan, Hai Zhao, Lei Li, and Jingjing Xu. 2023. Extrapolating Multilingual Understanding Models as Multilingual Generators. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 15432–15444, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Extrapolating Multilingual Understanding Models as Multilingual Generators (Wu et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/2023.findings-emnlp.1031.pdf