Interpretability for Morphological Inflection: from Character-level Predictions to Subword-level Rules
Tatyana Ruzsics, Olga Sozinova, Ximena Gutierrez-Vasques, Tanja Samardzic
Abstract
Neural models for morphological inflection have recently attained very high results. However, their interpretation remains challenging. Towards this goal, we propose a simple linguistically-motivated variant to the encoder-decoder model with attention. In our model, character-level cross-attention mechanism is complemented with a self-attention module over substrings of the input. We design a novel approach for pattern extraction from attention weights to interpret what the model learn. We apply our methodology to analyze the model’s decisions on three typologically-different languages and find that a) our pattern extraction method applied to cross-attention weights uncovers variation in form of inflection morphemes, b) pattern extraction from self-attention shows triggers for such variation, c) both types of patterns are closely aligned with grammar inflection classes and class assignment criteria, for all three languages. Additionally, we find that the proposed encoder attention component leads to consistent performance improvements over a strong baseline.- Anthology ID:
- 2021.eacl-main.278
- Volume:
- Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume
- Month:
- April
- Year:
- 2021
- Address:
- Online
- Editors:
- Paola Merlo, Jorg Tiedemann, Reut Tsarfaty
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 3189–3201
- Language:
- URL:
- https://aclanthology.org/2021.eacl-main.278
- DOI:
- 10.18653/v1/2021.eacl-main.278
- Cite (ACL):
- Tatyana Ruzsics, Olga Sozinova, Ximena Gutierrez-Vasques, and Tanja Samardzic. 2021. Interpretability for Morphological Inflection: from Character-level Predictions to Subword-level Rules. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3189–3201, Online. Association for Computational Linguistics.
- Cite (Informal):
- Interpretability for Morphological Inflection: from Character-level Predictions to Subword-level Rules (Ruzsics et al., EACL 2021)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/2021.eacl-main.278.pdf
- Code
- tatyana-ruzsics/interpretable-inflection