Abstract
In the persona-grounded dialogue (PGD) task, it is required not only to respond fluently, but also to ground the attributes according to the current conversation topic properly. However, due to their tendency to overly ground given attributes, LLMs often generate unnatural responses provoked by using attributes that deviate from the flow of the conversation or by exploiting too many attributes at once. We term this phenomenon the *overuse* problem of LLMs. Unfortunately, research devising precise criteria and frameworks to quantitatively verify LLMs’ *overuse* problem is obviously insufficient. To address this issue, we propose **P**ersona **A**ttributes **N**avigation for **D**etecting and **A**lleviating the *overuse* problem (**PANDA**) framework. **PANDA** is the first study to quantify the persona *overuse* problem of LLMs by establishing clear standards of the problem and verifying various LLMs based on them. Moreover, this framework navigates us into understanding persona attributes by introducing diverse and detailed dialogue topics that consider practical conversation situations. We provide insights related to LLMs’ persona attribute *overuse* problem through comprehensive verification and analysis with **PANDA** in the PGD task. Our code and resources can be found at http://github.com/jin62304/PANDA.- Anthology ID:
- 2024.emnlp-main.670
- Volume:
- Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
- Month:
- November
- Year:
- 2024
- Address:
- Miami, Florida, USA
- Editors:
- Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 12005–12026
- Language:
- URL:
- https://preview.aclanthology.org/add_missing_videos/2024.emnlp-main.670/
- DOI:
- 10.18653/v1/2024.emnlp-main.670
- Cite (ACL):
- Jinsung Kim, Seonmin Koo, and Heuiseok Lim. 2024. PANDA: Persona Attributes Navigation for Detecting and Alleviating Overuse Problem in Large Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 12005–12026, Miami, Florida, USA. Association for Computational Linguistics.
- Cite (Informal):
- PANDA: Persona Attributes Navigation for Detecting and Alleviating Overuse Problem in Large Language Models (Kim et al., EMNLP 2024)
- PDF:
- https://preview.aclanthology.org/add_missing_videos/2024.emnlp-main.670.pdf