Towards In-Context Non-Expert Evaluation of Reflection Generation for Counselling Conversations
Zixiu Wu, Simone Balloccu, Rim Helaoui, Diego Reforgiato Recupero, Daniele Riboni
Abstract
Reflection is an essential counselling strategy, where the therapist listens actively and responds with their own interpretation of the client’s words. Recent work leveraged pre-trained language models (PLMs) to approach reflection generation as a promising tool to aid counsellor training. However, those studies used limited dialogue context for modelling and simplistic error analysis for human evaluation. In this work, we take the first step towards addressing those limitations. First, we fine-tune PLMs on longer dialogue contexts for reflection generation. Then, we collect free-text error descriptions from non-experts about generated reflections, identify common patterns among them, and accordingly establish discrete error categories using thematic analysis. Based on this scheme, we plan for future work a mass non-expert error annotation phase for generated reflections followed by an expert-based validation phase, namely “whether a coherent and consistent response is a good reflection”.- Anthology ID:
- 2022.gem-1.9
- Volume:
- Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM)
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates (Hybrid)
- Editors:
- Antoine Bosselut, Khyathi Chandu, Kaustubh Dhole, Varun Gangal, Sebastian Gehrmann, Yacine Jernite, Jekaterina Novikova, Laura Perez-Beltrachini
- Venue:
- GEM
- SIG:
- SIGGEN
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 116–124
- Language:
- URL:
- https://aclanthology.org/2022.gem-1.9
- DOI:
- 10.18653/v1/2022.gem-1.9
- Cite (ACL):
- Zixiu Wu, Simone Balloccu, Rim Helaoui, Diego Reforgiato Recupero, and Daniele Riboni. 2022. Towards In-Context Non-Expert Evaluation of Reflection Generation for Counselling Conversations. In Proceedings of the 2nd Workshop on Natural Language Generation, Evaluation, and Metrics (GEM), pages 116–124, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
- Cite (Informal):
- Towards In-Context Non-Expert Evaluation of Reflection Generation for Counselling Conversations (Wu et al., GEM 2022)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2022.gem-1.9.pdf