DELPHI: Data for Evaluating LLMs’ Performance in Handling Controversial Issues
David Sun, Artem Abzaliev, Hadas Kotek, Christopher Klein, Zidi Xiu, Jason Williams
Abstract
Controversy is a reflection of our zeitgeist, and an important aspect to any discourse. The rise of large language models (LLMs) as conversational systems has increased public reliance on these systems for answers to their various questions. Consequently, it is crucial to systematically examine how these models respond to questions that pertaining to ongoing debates. However, few such datasets exist in providing human-annotated labels reflecting the contemporary discussions. To foster research in this area, we propose a novel construction of a controversial questions dataset, expanding upon the publicly released Quora Question Pairs Dataset. This dataset presents challenges concerning knowledge recency, safety, fairness, and bias. We evaluate different LLMs using a subset of this dataset, illuminating how they handle controversial issues and the stances they adopt. This research ultimately contributes to our understanding of LLMs’ interaction with controversial issues, paving the way for improvements in their comprehension and handling of complex societal debates.- Anthology ID:
- 2023.emnlp-industry.76
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Mingxuan Wang, Imed Zitouni
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 820–827
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-industry.76
- DOI:
- 10.18653/v1/2023.emnlp-industry.76
- Cite (ACL):
- David Sun, Artem Abzaliev, Hadas Kotek, Christopher Klein, Zidi Xiu, and Jason Williams. 2023. DELPHI: Data for Evaluating LLMs’ Performance in Handling Controversial Issues. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 820–827, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- DELPHI: Data for Evaluating LLMs’ Performance in Handling Controversial Issues (Sun et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/corrections-2024-05/2023.emnlp-industry.76.pdf