Abstract
We propose a method of machine-assisted annotation for the identification of tension development, annotating whether the tension is increasing, decreasing, or staying unchanged. We use a neural network based prediction model, whose predicted results are given to the annotators as initial values for the options that they are asked to choose. By presenting such initial values to the annotators, the annotation task becomes an evaluation task where the annotators inspect whether or not the predicted results are correct. To demonstrate the effectiveness of our method, we performed the annotation task in both in-house and crowdsourced environments. For the crowdsourced environment, we compared the annotation results with and without our method of machine-assisted annotation. We find that the results with our method showed a higher agreement to the gold standard than those without, though our method had little effect at reducing the time for annotation. Our codes for the experiment are made publicly available.- Anthology ID:
- D19-5906
- Volume:
- Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP
- Month:
- November
- Year:
- 2019
- Address:
- Hong Kong
- Editors:
- Silviu Paun, Dirk Hovy
- Venue:
- WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 39–47
- Language:
- URL:
- https://aclanthology.org/D19-5906
- DOI:
- 10.18653/v1/D19-5906
- Cite (ACL):
- Seungwon Yoon, Wonsuk Yang, and Jong Park. 2019. Computer Assisted Annotation of Tension Development in TED Talks through Crowdsourcing. In Proceedings of the First Workshop on Aggregating and Analysing Crowdsourced Annotations for NLP, pages 39–47, Hong Kong. Association for Computational Linguistics.
- Cite (Informal):
- Computer Assisted Annotation of Tension Development in TED Talks through Crowdsourcing (Yoon et al., 2019)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/D19-5906.pdf