Abstract
This paper describes a crowdsourcing experiment on the annotation of plot-like structures in English news articles. CrowdThruth methodology and metrics have been applied to select valid annotations from the crowd. We further run an in-depth analysis of the annotated data by comparing them with available expert data. Our results show a valuable use of crowdsourcing annotations for such complex semantic tasks, and suggest a new annotation approach which combine crowd and experts.- Anthology ID:
- W18-4306
- Volume:
- Proceedings of the Workshop Events and Stories in the News 2018
- Month:
- August
- Year:
- 2018
- Address:
- Santa Fe, New Mexico, U.S.A
- Editors:
- Tommaso Caselli, Ben Miller, Marieke van Erp, Piek Vossen, Martha Palmer, Eduard Hovy, Teruko Mitamura, David Caswell, Susan W. Brown, Claire Bonial
- Venue:
- EventStory
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 44–54
- Language:
- URL:
- https://aclanthology.org/W18-4306
- DOI:
- Cite (ACL):
- Tommaso Caselli and Oana Inel. 2018. Crowdsourcing StoryLines: Harnessing the Crowd for Causal Relation Annotation. In Proceedings of the Workshop Events and Stories in the News 2018, pages 44–54, Santa Fe, New Mexico, U.S.A. Association for Computational Linguistics.
- Cite (Informal):
- Crowdsourcing StoryLines: Harnessing the Crowd for Causal Relation Annotation (Caselli & Inel, EventStory 2018)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/W18-4306.pdf
- Code
- CrowdTruth/Crowdsourcing-StoryLines
- Data
- FrameNet