Abstract
Emphasis is a crucial component in human communication, which indicates speaker’s intention and implication beyond pure text in dialogue. While Large Language Models (LLMs) have revolutionized natural language processing, their ability to understand emphasis in dialogue remains uncertain. This paper introduces Emphasized-Talk, a benchmark dataset with annotated dialogue samples capturing the implications of emphasis. We evaluate various LLMs, both open-source and commercial, to assess their performance in understanding and generating emphasis. Additionally, we propose an automatic evaluation pipeline using GPT-4, which achieve high correlation with human scoring. Our findings reveal that although commercial LLMs generally perform better, there is still significant room for improvement in comprehending emphasized sentences.- Anthology ID:
- 2024.findings-emnlp.782
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2024
- Month:
- November
- Year:
- 2024
- Address:
- Miami, Florida, USA
- Editors:
- Yaser Al-Onaizan, Mohit Bansal, Yun-Nung Chen
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 13391–13401
- Language:
- URL:
- https://preview.aclanthology.org/add_missing_videos/2024.findings-emnlp.782/
- DOI:
- 10.18653/v1/2024.findings-emnlp.782
- Cite (ACL):
- Guan-Ting Lin and Hung-yi Lee. 2024. Can LLMs Understand the Implication of Emphasized Sentences in Dialogue?. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13391–13401, Miami, Florida, USA. Association for Computational Linguistics.
- Cite (Informal):
- Can LLMs Understand the Implication of Emphasized Sentences in Dialogue? (Lin & Lee, Findings 2024)
- PDF:
- https://preview.aclanthology.org/add_missing_videos/2024.findings-emnlp.782.pdf