Abstract
The recent paradigm shift to contextual word embeddings has seen tremendous success across a wide range of down-stream tasks. However, little is known on how the emergent relation of context and semantics manifests geometrically. We investigate polysemous words as one particularly prominent instance of semantic organization. Our rigorous quantitative analysis of linear separability and cluster organization in embedding vectors produced by BERT shows that semantics do not surface as isolated clusters but form seamless structures, tightly coupled with sentiment and syntax.- Anthology ID:
- 2020.blackboxnlp-1.15
- Volume:
- Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Venue:
- BlackboxNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 156–162
- Language:
- URL:
- https://aclanthology.org/2020.blackboxnlp-1.15
- DOI:
- 10.18653/v1/2020.blackboxnlp-1.15
- Cite (ACL):
- David Yenicelik, Florian Schmidt, and Yannic Kilcher. 2020. How does BERT capture semantics? A closer look at polysemous words. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 156–162, Online. Association for Computational Linguistics.
- Cite (Informal):
- How does BERT capture semantics? A closer look at polysemous words (Yenicelik et al., BlackboxNLP 2020)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2020.blackboxnlp-1.15.pdf