Computer Assisted Annotation of Tension Development in TED Talks through Crowdsourcing

Seungwon Yoon, Wonsuk Yang, Jong Park


Abstract
We propose a method of machine-assisted annotation for the identification of tension development, annotating whether the tension is increasing, decreasing, or staying unchanged. We use a neural network based prediction model, whose predicted results are given to the annotators as initial values for the options that they are asked to choose. By presenting such initial values to the annotators, the annotation task becomes an evaluation task where the annotators inspect whether or not the predicted results are correct. To demonstrate the effectiveness of our method, we performed the annotation task in both in-house and crowdsourced environments. For the crowdsourced environment, we compared the annotation results with and without our method of machine-assisted annotation. We find that the results with our method showed a higher agreement to the gold standard than those without, though our method had little effect at reducing the time for annotation. Our codes for the experiment are made publicly available.
Anthology ID:
D19-5906
Volume:
Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP
Month:
November
Year:
2019
Address:
Hong Kong
Venues:
EMNLP | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
39–47
Language:
URL:
https://aclanthology.org/D19-5906
DOI:
10.18653/v1/D19-5906
Bibkey:
Cite (ACL):
Seungwon Yoon, Wonsuk Yang, and Jong Park. 2019. Computer Assisted Annotation of Tension Development in TED Talks through Crowdsourcing. In Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP, pages 39–47, Hong Kong. Association for Computational Linguistics.
Cite (Informal):
Computer Assisted Annotation of Tension Development in TED Talks through Crowdsourcing (Yoon et al., EMNLP 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/update-css-js/D19-5906.pdf