Top-Rank-Focused Adaptive Vote Collection for the Evaluation of Domain-Specific Semantic Models
Pierangelo Lombardo, Alessio Boiardi, Luca Colombo, Angelo Schiavone, Nicolò Tamagnone
Abstract
The growth of domain-specific applications of semantic models, boosted by the recent achievements of unsupervised embedding learning algorithms, demands domain-specific evaluation datasets. In many cases, content-based recommenders being a prime example, these models are required to rank words or texts according to their semantic relatedness to a given concept, with particular focus on top ranks. In this work, we give a threefold contribution to address these requirements: (i) we define a protocol for the construction, based on adaptive pairwise comparisons, of a relatedness-based evaluation dataset tailored on the available resources and optimized to be particularly accurate in top-rank evaluation; (ii) we define appropriate metrics, extensions of well-known ranking correlation coefficients, to evaluate a semantic model via the aforementioned dataset by taking into account the greater significance of top ranks. Finally, (iii) we define a stochastic transitivity model to simulate semantic-driven pairwise comparisons, which confirms the effectiveness of the proposed dataset construction protocol.- Anthology ID:
- 2020.emnlp-main.249
- Volume:
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 3081–3093
- Language:
- URL:
- https://aclanthology.org/2020.emnlp-main.249
- DOI:
- 10.18653/v1/2020.emnlp-main.249
- Cite (ACL):
- Pierangelo Lombardo, Alessio Boiardi, Luca Colombo, Angelo Schiavone, and Nicolò Tamagnone. 2020. Top-Rank-Focused Adaptive Vote Collection for the Evaluation of Domain-Specific Semantic Models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3081–3093, Online. Association for Computational Linguistics.
- Cite (Informal):
- Top-Rank-Focused Adaptive Vote Collection for the Evaluation of Domain-Specific Semantic Models (Lombardo et al., EMNLP 2020)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2020.emnlp-main.249.pdf
- Code
- intervieweb-datascience/adaptive-comp