LLMs of Catan: Exploring Pragmatic Capabilities of Generative Chatbots Through Prediction and Classification of Dialogue Acts in Boardgames’ Multi-party Dialogues
Andrea Martinenghi, Gregor Donabauer, Simona Amenta, Sathya Bursic, Mathyas Giudici, Udo Kruschwitz, Franca Garzotto, Dimitri Ognibene
Abstract
Human language interactions involve complex processes beyond pure information exchange, for example, actions aimed at influencing beliefs and behaviors within a communicative context. In this paper, we propose to investigate the dialogue understanding capabilities of large language models (LLMs), particularly in multi-party settings, where challenges like speaker identification and turn-taking are common. Through experiments on the game-based STAC dataset, we explore zero and few-shot learning approaches for dialogue act classification in a multi-party game setting. Our intuition is that LLMs may excel in tasks framed through examples rather than formal descriptions, influenced by a range of pragmatic features like information presentation order in prompts and others. We also explore the models’ predictive abilities regarding future dialogue acts and study integrating information on dialogue act sequences to improve predictions. Our findings suggest that ChatGPT can keep up with baseline models trained from scratch for classification of certain dialogue act types but also reveal biases and limitations associated with the approach. These insights can be valuable for the development of multi-party chatbots and we try to point out directions for future research towards nuanced understanding and adaptation in diverse conversational contexts- Anthology ID:
- 2024.games-1.12
- Volume:
- Proceedings of the 10th Workshop on Games and Natural Language Processing @ LREC-COLING 2024
- Month:
- May
- Year:
- 2024
- Address:
- Torino, Italia
- Editors:
- Chris Madge, Jon Chamberlain, Karen Fort, Udo Kruschwitz, Stephanie Lukin
- Venues:
- games | WS
- SIG:
- Publisher:
- ELRA and ICCL
- Note:
- Pages:
- 107–118
- Language:
- URL:
- https://aclanthology.org/2024.games-1.12
- DOI:
- Cite (ACL):
- Andrea Martinenghi, Gregor Donabauer, Simona Amenta, Sathya Bursic, Mathyas Giudici, Udo Kruschwitz, Franca Garzotto, and Dimitri Ognibene. 2024. LLMs of Catan: Exploring Pragmatic Capabilities of Generative Chatbots Through Prediction and Classification of Dialogue Acts in Boardgames’ Multi-party Dialogues. In Proceedings of the 10th Workshop on Games and Natural Language Processing @ LREC-COLING 2024, pages 107–118, Torino, Italia. ELRA and ICCL.
- Cite (Informal):
- LLMs of Catan: Exploring Pragmatic Capabilities of Generative Chatbots Through Prediction and Classification of Dialogue Acts in Boardgames’ Multi-party Dialogues (Martinenghi et al., games-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.games-1.12.pdf