It’s All Fun and Games until Someone Annotates: Video Games with a Purpose for Linguistic Annotation

David Jurgens, Roberto Navigli


Abstract
Annotated data is prerequisite for many NLP applications. Acquiring large-scale annotated corpora is a major bottleneck, requiring significant time and resources. Recent work has proposed turning annotation into a game to increase its appeal and lower its cost; however, current games are largely text-based and closely resemble traditional annotation tasks. We propose a new linguistic annotation paradigm that produces annotations from playing graphical video games. The effectiveness of this design is demonstrated using two video games: one to create a mapping from WordNet senses to images, and a second game that performs Word Sense Disambiguation. Both games produce accurate results. The first game yields annotation quality equal to that of experts and a cost reduction of 73% over equivalent crowdsourcing; the second game provides a 16.3% improvement in accuracy over current state-of-the-art sense disambiguation games with WordNet.
Anthology ID:
Q14-1035
Volume:
Transactions of the Association for Computational Linguistics, Volume 2
Month:
Year:
2014
Address:
Cambridge, MA
Venue:
TACL
SIG:
Publisher:
MIT Press
Note:
Pages:
449–464
Language:
URL:
https://aclanthology.org/Q14-1035
DOI:
10.1162/tacl_a_00195
Bibkey:
Cite (ACL):
David Jurgens and Roberto Navigli. 2014. It’s All Fun and Games until Someone Annotates: Video Games with a Purpose for Linguistic Annotation. Transactions of the Association for Computational Linguistics, 2:449–464.
Cite (Informal):
It’s All Fun and Games until Someone Annotates: Video Games with a Purpose for Linguistic Annotation (Jurgens & Navigli, TACL 2014)
Copy Citation:
PDF:
https://preview.aclanthology.org/update-css-js/Q14-1035.pdf
Video:
 https://vimeo.com/150290361
Data
ImageNet