Abstract
Recent work on evaluating representation learning architectures in NLP has established a need for evaluation protocols based on subconscious cognitive measures rather than manually tailored intrinsic similarity and relatedness tasks. In this work, we propose a novel evaluation framework that enables large-scale evaluation of such architectures in the free word association (WA) task, which is firmly grounded in cognitive theories of human semantic representation. This evaluation is facilitated by the existence of large manually constructed repositories of word association data. In this paper, we (1) present a detailed analysis of the new quantitative WA evaluation protocol, (2) suggest new evaluation metrics for the WA task inspired by its direct analogy with information retrieval problems, (3) evaluate various state-of-the-art representation models on this task, and (4) discuss the relationship between WA and prior evaluations of semantic representation with well-known similarity and relatedness evaluation sets. We have made the WA evaluation toolkit publicly available.- Anthology ID:
- E17-1016
- Volume:
- Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers
- Month:
- April
- Year:
- 2017
- Address:
- Valencia, Spain
- Editors:
- Mirella Lapata, Phil Blunsom, Alexander Koller
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 163–175
- Language:
- URL:
- https://aclanthology.org/E17-1016
- DOI:
- Cite (ACL):
- Ivan Vulić, Douwe Kiela, and Anna Korhonen. 2017. Evaluation by Association: A Systematic Study of Quantitative Word Association Evaluation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 163–175, Valencia, Spain. Association for Computational Linguistics.
- Cite (Informal):
- Evaluation by Association: A Systematic Study of Quantitative Word Association Evaluation (Vulić et al., EACL 2017)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/E17-1016.pdf