Abstract
The outstanding performance of transformer-based language models on a great variety of NLP and NLU tasks has stimulated interest in exploration of their inner workings. Recent research has been primarily focused on higher-level and complex linguistic phenomena such as syntax, semantics, world knowledge and common-sense. The majority of the studies is anglocentric, and little remains known regarding other languages, specifically their morphosyntactic properties. To this end, our work presents Morph Call, a suite of 46 probing tasks for four Indo-European languages of different morphology: Russian, French, English and German. We propose a new type of probing tasks based on detection of guided sentence perturbations. We use a combination of neuron-, layer- and representation-level introspection techniques to analyze the morphosyntactic content of four multilingual transformers, including their understudied distilled versions. Besides, we examine how fine-tuning on POS-tagging task affects the probing performance.- Anthology ID:
- 2021.sigtyp-1.10
- Volume:
- Proceedings of the Third Workshop on Computational Typology and Multilingual NLP
- Month:
- June
- Year:
- 2021
- Address:
- Online
- Editors:
- Ekaterina Vylomova, Elizabeth Salesky, Sabrina Mielke, Gabriella Lapesa, Ritesh Kumar, Harald Hammarström, Ivan Vulić, Anna Korhonen, Roi Reichart, Edoardo Maria Ponti, Ryan Cotterell
- Venue:
- SIGTYP
- SIG:
- SIGTYP
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 97–121
- Language:
- URL:
- https://aclanthology.org/2021.sigtyp-1.10
- DOI:
- 10.18653/v1/2021.sigtyp-1.10
- Cite (ACL):
- Vladislav Mikhailov, Oleg Serikov, and Ekaterina Artemova. 2021. Morph Call: Probing Morphosyntactic Content of Multilingual Transformers. In Proceedings of the Third Workshop on Computational Typology and Multilingual NLP, pages 97–121, Online. Association for Computational Linguistics.
- Cite (Informal):
- Morph Call: Probing Morphosyntactic Content of Multilingual Transformers (Mikhailov et al., SIGTYP 2021)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/2021.sigtyp-1.10.pdf
- Code
- morphology-probing/morph-call
- Data
- Morph Call, Universal Dependencies