Improving Word Sense Induction through Adversarial Forgetting of Morphosyntactic Information
Deniz Yava, Timothée Bernard, Laura Kallmeyer, Benoît Crabbé
Abstract
This paper addresses the problem of word sense induction (WSI) via clustering of word embeddings. It starts from the hypothesis that contextualized word representations obtained from pre-trained language models (LMs), while being a valuable source for WSI, encode more information than what is necessary for the identification of word senses and some of this information affect the performance negatively in unsupervised settings. We investigate whether using contextualized representations that are invariant to these ‘nuisance features’ can increase WSI performance. For this purpose, we propose an adaptation of the adversarial training framework proposed by Jaiswal et al. (2020) to erase specific information from the representations of LMs, thereby creating feature-invariant representations. We experiment with erasing (i) morphological and (ii) syntactic features. The results of subsequent clustering for WSI show that these features indeed act like noise: Using feature-invariant representations, compared to using the original representations, increases clustering-based WSI performance. Furthermore, we provide an in-depth analysis of how the information about the syntactic and morphological features of words relate to and affect WSI performance.- Anthology ID:
- 2024.starsem-1.19
- Volume:
- Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024)
- Month:
- June
- Year:
- 2024
- Address:
- Mexico City, Mexico
- Editors:
- Danushka Bollegala, Vered Shwartz
- Venue:
- *SEM
- SIG:
- SIGLEX
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 238–251
- Language:
- URL:
- https://aclanthology.org/2024.starsem-1.19
- DOI:
- Cite (ACL):
- Deniz Yava, Timothée Bernard, Laura Kallmeyer, and Benoît Crabbé. 2024. Improving Word Sense Induction through Adversarial Forgetting of Morphosyntactic Information. In Proceedings of the 13th Joint Conference on Lexical and Computational Semantics (*SEM 2024), pages 238–251, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- Improving Word Sense Induction through Adversarial Forgetting of Morphosyntactic Information (Yava et al., *SEM 2024)
- PDF:
- https://preview.aclanthology.org/jeptaln-2024-ingestion/2024.starsem-1.19.pdf