Using ChatGPT for Annotation of Attitude within the Appraisal Theory: Lessons Learned
Mirela Imamovic, Silvana Deilen, Dylan Glynn, Ekaterina Lapshinova-Koltunski
Abstract
We investigate the potential of using ChatGPT to annotate complex linguistic phenomena, such as language of evaluation, attitude and emotion. For this, we automatically annotate 11 texts in English, which represent spoken popular science, and evaluate the annotations manually. Our results show that ChatGPT has good precision in itemisation, i.e. detecting linguistic items in the text that carry evaluative meaning. However, we also find that the recall is very low. Besides that, we state that the tool fails in labeling the detected items with the correct categories on a more fine-grained level of granularity. We analyse the errors to find systematic errors related to specific categories in the annotation scheme.- Anthology ID:
- 2024.law-1.11
- Volume:
- Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII)
- Month:
- March
- Year:
- 2024
- Address:
- St. Julians, Malta
- Editors:
- Sophie Henning, Manfred Stede
- Venues:
- LAW | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 112–123
- Language:
- URL:
- https://aclanthology.org/2024.law-1.11
- DOI:
- Cite (ACL):
- Mirela Imamovic, Silvana Deilen, Dylan Glynn, and Ekaterina Lapshinova-Koltunski. 2024. Using ChatGPT for Annotation of Attitude within the Appraisal Theory: Lessons Learned. In Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII), pages 112–123, St. Julians, Malta. Association for Computational Linguistics.
- Cite (Informal):
- Using ChatGPT for Annotation of Attitude within the Appraisal Theory: Lessons Learned (Imamovic et al., LAW-WS 2024)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2024.law-1.11.pdf