Abstract
Annotation quality is often framed as post-hoc cleanup of annotator-caused issues. This position paper discusses whether, how, and why this narrative limits the scope of improving annotation. We call to consider annotation as a procedural collaboration, outlining three points in this direction:(1) An issue can be either annotator- or researcher-oriented, where one party is accountable and the other party may lack ability to fix it; (2) yet, they can co-occur or have similar consequences, and thus any specific problem we encounter may be a combination;(3) therefore, we need a new language to capture the nuance and holistically describe the full procedure to resolve these issues.To that end, we propose to study how agency is manifested in annotation and picture how this perspective benefits the community more broadly.- Anthology ID:
- 2024.findings-acl.518
- Volume:
- Findings of the Association for Computational Linguistics ACL 2024
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand and virtual meeting
- Editors:
- Lun-Wei Ku, Andre Martins, Vivek Srikumar
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 8773–8782
- Language:
- URL:
- https://aclanthology.org/2024.findings-acl.518
- DOI:
- Cite (ACL):
- Shengqi Zhu and Jeffrey Rzeszotarski. 2024. “Get Their Hands Dirty, Not Mine”: On Researcher-Annotator Collaboration and the Agency of Annotators. In Findings of the Association for Computational Linguistics ACL 2024, pages 8773–8782, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
- Cite (Informal):
- “Get Their Hands Dirty, Not Mine”: On Researcher-Annotator Collaboration and the Agency of Annotators (Zhu & Rzeszotarski, Findings 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.findings-acl.518.pdf