Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation

Kun Zhou, Yifan Li, Xin Zhao, Ji-Rong Wen

Abstract
Recently, continuous diffusion models (CDM) have been introduced into non-autoregressive (NAR) text-to-text generation. However, the discrete nature of text increases the difficulty of CDM to generate coherent and fluent texts, and also causes the incompatibility problem between CDM and advanced NLP techniques, especially the popular pre-trained language models (PLMs).To solve it, we propose Diffusion-NAT, which introduces discrete diffusion models (DDM) into NAR text-to-text generation and integrates BART to improve the performance.By revising the decoding process of BART and the typical settings of DDM, we unify the inference process of BART and the denoising process of DDM into the same NAR masked tokens recovering task.In this way, DDM can rely on BART to perform denoising, which can benefit from both the rich pre-learned knowledge of BART and the iterative refining paradigm of DDM.Besides, we also propose the iterative self-prompting strategy to further improve the generation quality.Experimental results on 7 datasets show that our approach can outperform competitive NAR methods, and even surpass autoregressive methods.Our code and data are released at https://github.com/RUCAIBox/DiffusionNAT.
Anthology ID:
2024.eacl-long.86
Volume:
Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)
Month:
March
Year:
2024
Address:
St. Julian’s, Malta
Editors:
Yvette Graham, Matthew Purver
Venue:
EACL
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1438–1451
Language:
URL:
https://aclanthology.org/2024.eacl-long.86
DOI:
Bibkey:
Cite (ACL):
Kun Zhou, Yifan Li, Xin Zhao, and Ji-Rong Wen. 2024. Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1438–1451, St. Julian’s, Malta. Association for Computational Linguistics.
Cite (Informal):
Diffusion-NAT: Self-Prompting Discrete Diffusion for Non-Autoregressive Text Generation (Zhou et al., EACL 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/emnlp22-frontmatter/2024.eacl-long.86.pdf
PDF Search