To Aggregate or Not to Aggregate. That is the Question: A Case Study on Annotation Subjectivity in Span Prediction
Kemal Kurniawan, Meladel Mistica, Timothy Baldwin, Jey Han Lau
Abstract
This paper explores the task of automatic prediction of text spans in a legal problem description that support a legal area label. We use a corpus of problem descriptions written by laypeople in English that is annotated by practising lawyers. Inherent subjectivity exists in our task because legal area categorisation is a complex task, and lawyers often have different views on a problem. Experiments show that training on majority-voted spans outperforms training on disaggregated ones.- Anthology ID:
- 2024.wassa-1.29
- Volume:
- Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Orphée De Clercq, Valentin Barriere, Jeremy Barnes, Roman Klinger, João Sedoc, Shabnam Tafreshi
- Venues:
- WASSA | WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 362–368
- Language:
- URL:
- https://aclanthology.org/2024.wassa-1.29
- DOI:
- Cite (ACL):
- Kemal Kurniawan, Meladel Mistica, Timothy Baldwin, and Jey Han Lau. 2024. To Aggregate or Not to Aggregate. That is the Question: A Case Study on Annotation Subjectivity in Span Prediction. In Proceedings of the 14th Workshop on Computational Approaches to Subjectivity, Sentiment, & Social Media Analysis, pages 362–368, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- To Aggregate or Not to Aggregate. That is the Question: A Case Study on Annotation Subjectivity in Span Prediction (Kurniawan et al., WASSA-WS 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.wassa-1.29.pdf