Controlling the Imprint of Passivization and Negation in Contextualized Representations

Hande Celikkanat, Sami Virpioja, Jörg Tiedemann, Marianna Apidianaki

Abstract
Contextualized word representations encode rich information about syntax and semantics, alongside specificities of each context of use. While contextual variation does not always reflect actual meaning shifts, it can still reduce the similarity of embeddings for word instances having the same meaning. We explore the imprint of two specific linguistic alternations, namely passivization and negation, on the representations generated by neural models trained with two different objectives: masked language modeling and translation. Our exploration methodology is inspired by an approach previously proposed for removing societal biases from word vectors. We show that passivization and negation leave their traces on the representations, and that neutralizing this information leads to more similar embeddings for words that should preserve their meaning in the transformation. We also find clear differences in how the respective features generalize across datasets.
Anthology ID:
2020.blackboxnlp-1.13
Volume:
Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP
Month:
November
Year:
2020
Address:
Online
Venue:
BlackboxNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
136–148
Language:
URL:
https://aclanthology.org/2020.blackboxnlp-1.13
DOI:
10.18653/v1/2020.blackboxnlp-1.13
Bibkey:
Cite (ACL):
Hande Celikkanat, Sami Virpioja, Jörg Tiedemann, and Marianna Apidianaki. 2020. Controlling the Imprint of Passivization and Negation in Contextualized Representations. In Proceedings of the Third BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 136–148, Online. Association for Computational Linguistics.
Cite (Informal):
Controlling the Imprint of Passivization and Negation in Contextualized Representations (Celikkanat et al., BlackboxNLP 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2020.blackboxnlp-1.13.pdf
Data
SICK
PDF Search