Abstract
Diffusion models have garnered considerable interest in the field of text generation. Several studies have explored text diffusion models with different structures and applied them to various tasks, including named entity recognition and summarization. However, there exists a notable disparity between the “easy-first” text generation process of current diffusion models and the “keyword-first” natural text generation process of humans, which has received limited attention. To bridge this gap, we propose InfoDiffusion, a non-autoregressive text diffusion model. Our approach introduces a “keyinfo-first” generation strategy and incorporates a noise schedule based on the amount of text information. In addition, InfoDiffusion combines self-conditioning with a newly proposed partially noising model structure. Experimental results show that InfoDiffusion outperforms the baseline model in terms of generation quality and diversity, as well as exhibiting higher sampling efficiency.- Anthology ID:
- 2023.findings-emnlp.919
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 13757–13770
- Language:
- URL:
- https://aclanthology.org/2023.findings-emnlp.919
- DOI:
- 10.18653/v1/2023.findings-emnlp.919
- Cite (ACL):
- Renzhi Wang, Jing Li, and Piji Li. 2023. InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13757–13770, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generation (Wang et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2023.findings-emnlp.919.pdf