Abstract
In this work, we address the evaluation of distributional semantic models trained on smaller, domain-specific texts, specifically, philosophical text. Specifically, we inspect the behaviour of models using a pre-trained background space in learning. We propose a measure of consistency which can be used as an evaluation metric when no in-domain gold-standard data is available. This measure simply computes the ability of a model to learn similar embeddings from different parts of some homogeneous data. We show that in spite of being a simple evaluation, consistency actually depends on various combinations of factors, including the nature of the data itself, the model used to train the semantic space, and the frequency of the learnt terms, both in the background space and in the in-domain data of interest.- Anthology ID:
- R19-1016
- Volume:
- Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019)
- Month:
- September
- Year:
- 2019
- Address:
- Varna, Bulgaria
- Editors:
- Ruslan Mitkov, Galia Angelova
- Venue:
- RANLP
- SIG:
- Publisher:
- INCOMA Ltd.
- Note:
- Pages:
- 132–141
- Language:
- URL:
- https://aclanthology.org/R19-1016
- DOI:
- 10.26615/978-954-452-056-4_016
- Cite (ACL):
- Jelke Bloem, Antske Fokkens, and Aurélie Herbelot. 2019. Evaluating the Consistency of Word Embeddings from Small Data. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2019), pages 132–141, Varna, Bulgaria. INCOMA Ltd..
- Cite (Informal):
- Evaluating the Consistency of Word Embeddings from Small Data (Bloem et al., RANLP 2019)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/R19-1016.pdf