John Funchion


2024

pdf
Are You Serious? Handling Disagreement When Annotating Conspiracy Theory Texts
Ashley Hemm | Sandra Kübler | Michelle Seelig | John Funchion | Manohar Murthi | Kamal Premaratne | Daniel Verdear | Stefan Wuchty
Proceedings of The 18th Linguistic Annotation Workshop (LAW-XVIII)

We often assume that annotation tasks, such as annotating for the presence of conspiracy theories, can be annotated with hard labels, without definitions or guidelines. Our annotation experiments, comparing students and experts, show that there is little agreement on basic annotations even among experts. For this reason, we conclude that we need to accept disagreement as an integral part of such annotations.