Abstract
Pretrained language models (PLMs), especially large language models (LLMs) demonstrate impressive capabilities in open-ended text generation. While our statistical results show that LLMs often suffer from over-concentrated information, where the generated texts overly focus on the given prompt and fail to provide sufficient background and detailed information as humans do. To address this issue, we propose a dynamic knowledge-guided informative open-ended text generation approach, that utilizes a knowledge graph to help the model generate more contextually related entities and detailed facts. Specifically, we first employ a local knowledge filter to extract relevant knowledge from the comprehensive knowledge graph for a given topic sentence. Then we introduce a dynamic knowledge selector to predict the entity to be mentioned in the subsequent sentence. Finally, we utilize a knowledge-enhanced text generator to produce a more informative output. To evaluate the effectiveness of our approach, we evaluate the proposed approach in two scenarios: fine-tuning for small PLMs and prompt tuning for LLMs. Experimental results show that our approach could generate more informative texts than baselines.- Anthology ID:
- 2023.findings-emnlp.210
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 3189–3203
- Language:
- URL:
- https://aclanthology.org/2023.findings-emnlp.210
- DOI:
- 10.18653/v1/2023.findings-emnlp.210
- Cite (ACL):
- Zixuan Ren, Yang Zhao, and Chengqing Zong. 2023. Towards Informative Open-ended Text Generation with Dynamic Knowledge Triples. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3189–3203, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Towards Informative Open-ended Text Generation with Dynamic Knowledge Triples (Ren et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2023.findings-emnlp.210.pdf