Abstract
Sparse word embeddings models (SPINE, SINr) are designed to embed words in interpretable dimensions. An interpretable dimension is such that a human can interpret the semantic (or syntactic) relations between words active for a dimension. These models are useful for critical downstream tasks in natural language processing (e.g. medical or legal NLP), and digital humanities applications. This work extends interpretability at the vector level with a more manageable number of activated dimensions following recommendations from psycholinguistics. Subsequently, one of the key criteria to an interpretable model is sparsity: in order to be interpretable, not every word should be represented by all the features of the model, especially if humans have to interpret these features and their relations. This raises one question: to which extent is sparsity sustainable with regard to performance? We thus introduce a sparsification procedure to evaluate its impact on two interpretable methods (SPINE and SINr) to tend towards sustainable vector interpretability. We also introduce stability as a new criterion to interpretability. Our stability evaluations show little albeit non-zero variation for SPINE and SINr embeddings. We then show that increasing sparsity does not necessarily interfere with performance. These results are encouraging and pave the way towards intrinsically interpretable word vectors.- Anthology ID:
- 2023.iwcs-1.13
- Volume:
- Proceedings of the 15th International Conference on Computational Semantics
- Month:
- June
- Year:
- 2023
- Address:
- Nancy, France
- Editors:
- Maxime Amblard, Ellen Breitholtz
- Venue:
- IWCS
- SIG:
- SIGSEM
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 106–115
- Language:
- URL:
- https://aclanthology.org/2023.iwcs-1.13
- DOI:
- Cite (ACL):
- Simon Guillot, Thibault Prouteau, and Nicolas Dugue. 2023. Sparser is better: one step closer to word embedding interpretability. In Proceedings of the 15th International Conference on Computational Semantics, pages 106–115, Nancy, France. Association for Computational Linguistics.
- Cite (Informal):
- Sparser is better: one step closer to word embedding interpretability (Guillot et al., IWCS 2023)
- PDF:
- https://preview.aclanthology.org/revert-3132-ingestion-checklist/2023.iwcs-1.13.pdf