Abstract
Downstream evaluation of pretrained word embeddings is expensive, more so for tasks where current state of the art models are very large architectures. Intrinsic evaluation using word similarity or analogy datasets, on the other hand, suffers from several disadvantages. We propose a novel intrinsic evaluation task employing large word association datasets (particularly the Small World of Words dataset). We observe correlations not just between performances on SWOW-8500 and previously proposed intrinsic tasks of word similarity prediction, but also with downstream tasks (eg. Text Classification and Natural Language Inference). Most importantly, we report better confidence intervals for scores on our word association task, with no fall in correlation with downstream performance.- Anthology ID:
- W19-2006
- Volume:
- Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP
- Month:
- June
- Year:
- 2019
- Address:
- Minneapolis, USA
- Editors:
- Anna Rogers, Aleksandr Drozd, Anna Rumshisky, Yoav Goldberg
- Venue:
- RepEval
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 43–51
- Language:
- URL:
- https://aclanthology.org/W19-2006
- DOI:
- 10.18653/v1/W19-2006
- Cite (ACL):
- Avijit Thawani, Biplav Srivastava, and Anil Singh. 2019. SWOW-8500: Word Association task for Intrinsic Evaluation of Word Embeddings. In Proceedings of the 3rd Workshop on Evaluating Vector Space Representations for NLP, pages 43–51, Minneapolis, USA. Association for Computational Linguistics.
- Cite (Informal):
- SWOW-8500: Word Association task for Intrinsic Evaluation of Word Embeddings (Thawani et al., RepEval 2019)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/W19-2006.pdf
- Code
- avi-jit/SWOW-eval
- Data
- ConceptNet