Abstract
Research on dialogue constructiveness assessment focuses on (i) analysing conversational factors that influence individuals to take specific actions, win debates, change their perspectives or broaden their open-mindedness and (ii) predicting constructiveness outcomes following dialogues for such use cases. These objectives can be achieved by training either interpretable feature-based models (which often involve costly human annotations) or neural models such as pre-trained language models (which have empirically shown higher task accuracy but lack interpretability). In this paper we propose an LLM feature-based framework for dialogue constructiveness assessment that combines the strengths of feature-based and neural approaches, while mitigating their downsides. The framework first defines a set of dataset-independent and interpretable linguistic features, which can be extracted by both prompting an LLM and simple heuristics. Such features are then used to train LLM feature-based models. We apply this framework to three datasets of dialogue constructiveness and find that our LLM feature-based models outperform or performs at least as well as standard feature-based models and neural models. We also find that the LLM feature-based model learns more robust prediction rules instead of relying on superficial shortcuts, which often trouble neural models.- Anthology ID:
- 2024.emnlp-main.308
- Volume:
- Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing
- Month:
- November
- Year:
- 2024
- Address:
- Miami, Florida, USA
- Editors:
- Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 5389–5409
- Language:
- URL:
- https://preview.aclanthology.org/add_missing_videos/2024.emnlp-main.308/
- DOI:
- 10.18653/v1/2024.emnlp-main.308
- Cite (ACL):
- Lexin Zhou, Youmna Farag, and Andreas Vlachos. 2024. An LLM Feature-based Framework for Dialogue Constructiveness Assessment. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5389–5409, Miami, Florida, USA. Association for Computational Linguistics.
- Cite (Informal):
- An LLM Feature-based Framework for Dialogue Constructiveness Assessment (Zhou et al., EMNLP 2024)
- PDF:
- https://preview.aclanthology.org/add_missing_videos/2024.emnlp-main.308.pdf