Sensing and Learning Human Annotators Engaged in Narrative Sensemaking
McKenna Tornblad, Luke Lapresi, Christopher Homan, Raymond Ptucha, Cecilia Ovesdotter Alm
Abstract
While labor issues and quality assurance in crowdwork are increasingly studied, how annotators make sense of texts and how they are personally impacted by doing so are not. We study these questions via a narrative-sorting annotation task, where carefully selected (by sequentiality, topic, emotional content, and length) collections of tweets serve as examples of everyday storytelling. As readers process these narratives, we measure their facial expressions, galvanic skin response, and self-reported reactions. From the perspective of annotator well-being, a reassuring outcome was that the sorting task did not cause a measurable stress response, however readers reacted to humor. In terms of sensemaking, readers were more confident when sorting sequential, target-topical, and highly emotional tweets. As crowdsourcing becomes more common, this research sheds light onto the perceptive capabilities and emotional impact of human readers.- Anthology ID:
- N18-4019
- Volume:
- Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop
- Month:
- June
- Year:
- 2018
- Address:
- New Orleans, Louisiana, USA
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 136–143
- Language:
- URL:
- https://aclanthology.org/N18-4019
- DOI:
- 10.18653/v1/N18-4019
- Cite (ACL):
- McKenna Tornblad, Luke Lapresi, Christopher Homan, Raymond Ptucha, and Cecilia Ovesdotter Alm. 2018. Sensing and Learning Human Annotators Engaged in Narrative Sensemaking. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 136–143, New Orleans, Louisiana, USA. Association for Computational Linguistics.
- Cite (Informal):
- Sensing and Learning Human Annotators Engaged in Narrative Sensemaking (Tornblad et al., NAACL 2018)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/N18-4019.pdf