Abstract
The overall goal of this project is to evaluate the performance of word sense alignment (WSA) systems, focusing on obtaining examples appropriate to language learners. Building a gold standard dataset based on human expert judgments is costly in time and labor, and thus we gauge the utility of using semi-experts in performing the annotation. In an online survey, we present a sense of a target word from one dictionary with senses from the other dictionary, asking for judgments of relatedness. We note the difficulty of agreement, yet the utility in using such results to evaluate WSA work. We find that one's treatment of related senses heavily impacts the results for WSA.- Anthology ID:
- L12-1380
- Volume:
- Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12)
- Month:
- May
- Year:
- 2012
- Address:
- Istanbul, Turkey
- Venue:
- LREC
- SIG:
- Publisher:
- European Language Resources Association (ELRA)
- Note:
- Pages:
- 605–611
- Language:
- URL:
- http://www.lrec-conf.org/proceedings/lrec2012/pdf/652_Paper.pdf
- DOI:
- Cite (ACL):
- Soojeong Eom, Markus Dickinson, and Graham Katz. 2012. Using semi-experts to derive judgments on word sense alignment: a pilot study. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 605–611, Istanbul, Turkey. European Language Resources Association (ELRA).
- Cite (Informal):
- Using semi-experts to derive judgments on word sense alignment: a pilot study (Eom et al., LREC 2012)
- PDF:
- http://www.lrec-conf.org/proceedings/lrec2012/pdf/652_Paper.pdf