Morph Call: Probing Morphosyntactic Content of Multilingual Transformers

Vladislav Mikhailov, Oleg Serikov, Ekaterina Artemova


Abstract
The outstanding performance of transformer-based language models on a great variety of NLP and NLU tasks has stimulated interest in exploration of their inner workings. Recent research has been primarily focused on higher-level and complex linguistic phenomena such as syntax, semantics, world knowledge and common-sense. The majority of the studies is anglocentric, and little remains known regarding other languages, specifically their morphosyntactic properties. To this end, our work presents Morph Call, a suite of 46 probing tasks for four Indo-European languages of different morphology: Russian, French, English and German. We propose a new type of probing tasks based on detection of guided sentence perturbations. We use a combination of neuron-, layer- and representation-level introspection techniques to analyze the morphosyntactic content of four multilingual transformers, including their understudied distilled versions. Besides, we examine how fine-tuning on POS-tagging task affects the probing performance.
Anthology ID:
2021.sigtyp-1.10
Volume:
Proceedings of the Third Workshop on Computational Typology and Multilingual NLP
Month:
June
Year:
2021
Address:
Online
Editors:
Ekaterina Vylomova, Elizabeth Salesky, Sabrina Mielke, Gabriella Lapesa, Ritesh Kumar, Harald Hammarström, Ivan Vulić, Anna Korhonen, Roi Reichart, Edoardo Maria Ponti, Ryan Cotterell
Venue:
SIGTYP
SIG:
SIGTYP
Publisher:
Association for Computational Linguistics
Note:
Pages:
97–121
Language:
URL:
https://aclanthology.org/2021.sigtyp-1.10
DOI:
10.18653/v1/2021.sigtyp-1.10
Bibkey:
Cite (ACL):
Vladislav Mikhailov, Oleg Serikov, and Ekaterina Artemova. 2021. Morph Call: Probing Morphosyntactic Content of Multilingual Transformers. In Proceedings of the Third Workshop on Computational Typology and Multilingual NLP, pages 97–121, Online. Association for Computational Linguistics.
Cite (Informal):
Morph Call: Probing Morphosyntactic Content of Multilingual Transformers (Mikhailov et al., SIGTYP 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-1/2021.sigtyp-1.10.pdf
Code
 morphology-probing/morph-call
Data
Morph CallUniversal Dependencies