Travis Rood


2010

pdf
Twitter in Mass Emergency: What NLP Can Contribute
William J. Corvey | Sarah Vieweg | Travis Rood | Martha Palmer
Proceedings of the NAACL HLT 2010 Workshop on Computational Linguistics in a World of Social Media

pdf
Number or Nuance: Which Factors Restrict Reliable Word Sense Annotation?
Susan Windisch Brown | Travis Rood | Martha Palmer
Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC'10)

This study attempts to pinpoint the factors that restrict reliable word sense annotation, focusing on the influence of the number of senses annotators use and the semantic granularity of those senses. Both of these factors may be possible causes of low interannotator agreement (ITA) when tagging with fine-grained word senses, and, consequently, low WSD system performance (Ng et al., 1999; Snyder & Palmer, 2004; Chklovski & Mihalcea, 2002). If number of senses is the culprit, modifying the task to show fewer senses at a time could improve annotator reliability. However, if overly nuanced distinctions are the problem, then more general, coarse-grained distinctions may be necessary for annotator success and may be all that is needed to supply systems with the types of distinctions that people make. We describe three experiments that explore the role of these factors in annotation performance. Our results indicate that of these two factors, only the granularity of the senses restricts interannotator agreement, with broader senses resulting in higher annotation reliability.