Abstract
Understanding empathy in text dialogue data is a difficult, yet critical, skill for effective human-machine interaction. In this work, we ask whether systems are making meaningful progress on this challenge. We consider a simple model that checks if an input utterance is similar to a small set of empathetic examples. Crucially, the model does not look at what the utterance is a response to, i.e., the dialogue context. This model performs comparably to other work on standard benchmarks and even outperforms state-of-the-art models for empathetic rationale extraction by 16.7 points on T-F1 and 4.3 on IOU-F1. This indicates that current systems rely on the surface form of the response, rather than whether it is suitable in context. To confirm this, we create examples with dialogue contexts that change the interpretation of the response and show that current systems continue to label utterances as empathetic. We discuss the implications of our findings, including improvements for empathetic benchmarks and how our model can be an informative baseline.- Anthology ID:
- 2023.eacl-main.123
- Original:
- 2023.eacl-main.123v1
- Version 2:
- 2023.eacl-main.123v2
- Volume:
- Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Andreas Vlachos, Isabelle Augenstein
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1686–1695
- Language:
- URL:
- https://aclanthology.org/2023.eacl-main.123
- DOI:
- 10.18653/v1/2023.eacl-main.123
- Cite (ACL):
- Andrew Lee, Jonathan K. Kummerfeld, Larry An, and Rada Mihalcea. 2023. Empathy Identification Systems are not Accurately Accounting for Context. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1686–1695, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- Empathy Identification Systems are not Accurately Accounting for Context (Lee et al., EACL 2023)
- PDF:
- https://preview.aclanthology.org/revert-3132-ingestion-checklist/2023.eacl-main.123.pdf