Abstract
Several studies investigated the linguistic information implicitly encoded in Neural Language Models. Most of these works focused on quantifying the amount and type of information available within their internal representations and across their layers. In line with this scenario, we proposed a different study, based on Lasso regression, aimed at understanding how the information encoded by BERT sentence-level representations is arrange within its hidden units. Using a suite of several probing tasks, we showed the existence of a relationship between the implicit knowledge learned by the model and the number of individual units involved in the encodings of this competence. Moreover, we found that it is possible to identify groups of hidden units more relevant for specific linguistic properties.- Anthology ID:
- 2021.deelio-1.6
- Volume:
- Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures
- Month:
- June
- Year:
- 2021
- Address:
- Online
- Editors:
- Eneko Agirre, Marianna Apidianaki, Ivan Vulić
- Venue:
- DeeLIO
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 48–57
- Language:
- URL:
- https://aclanthology.org/2021.deelio-1.6
- DOI:
- 10.18653/v1/2021.deelio-1.6
- Cite (ACL):
- Giovanni Puccetti, Alessio Miaschi, and Felice Dell’Orletta. 2021. How Do BERT Embeddings Organize Linguistic Knowledge?. In Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pages 48–57, Online. Association for Computational Linguistics.
- Cite (Informal):
- How Do BERT Embeddings Organize Linguistic Knowledge? (Puccetti et al., DeeLIO 2021)
- PDF:
- https://preview.aclanthology.org/add_acl24_videos/2021.deelio-1.6.pdf
- Data
- Universal Dependencies