Multilingual, Multi-scale and Multi-layer Visualization of Intermediate Representations

Carlos Escolano, Marta R. Costa-jussà, Elora Lacroux, Pere-Pau Vázquez


Abstract
The main alternatives nowadays to deal with sequences are Recurrent Neural Networks (RNN) architectures and the Transformer. In this context, Both RNN’s and Transformer have been used as an encoder-decoder architecture with multiple layers in each module. Far beyond this, these architectures are the basis for the contextual word embeddings which are revolutionizing most natural language downstream applications. However, intermediate representations in either the RNN or Transformer architectures can be difficult to interpret. To make these layer representations more accessible and meaningful, we introduce a web-based tool that visualizes them both at the sentence and token level. We present three use cases. The first analyses gender issues in contextual word embeddings. The second and third are showing multilingual intermediate representations for sentences and tokens and the evolution of these intermediate representations along with the multiple layers of the decoder and in the context of multilingual machine translation.
Anthology ID:
D19-3026
Volume:
Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations
Month:
November
Year:
2019
Address:
Hong Kong, China
Editors:
Sebastian Padó, Ruihong Huang
Venues:
EMNLP | IJCNLP
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
151–156
Language:
URL:
https://aclanthology.org/D19-3026
DOI:
10.18653/v1/D19-3026
Bibkey:
Cite (ACL):
Carlos Escolano, Marta R. Costa-jussà, Elora Lacroux, and Pere-Pau Vázquez. 2019. Multilingual, Multi-scale and Multi-layer Visualization of Intermediate Representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 151–156, Hong Kong, China. Association for Computational Linguistics.
Cite (Informal):
Multilingual, Multi-scale and Multi-layer Visualization of Intermediate Representations (Escolano et al., EMNLP-IJCNLP 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-bitext-workshop/D19-3026.pdf