Multilingual, Multi-scale and Multi-layer Visualization of Intermediate Representations
Carlos Escolano, Marta R. Costa-jussà, Elora Lacroux, Pere-Pau Vázquez
Abstract
The main alternatives nowadays to deal with sequences are Recurrent Neural Networks (RNN) architectures and the Transformer. In this context, Both RNN’s and Transformer have been used as an encoder-decoder architecture with multiple layers in each module. Far beyond this, these architectures are the basis for the contextual word embeddings which are revolutionizing most natural language downstream applications. However, intermediate representations in either the RNN or Transformer architectures can be difficult to interpret. To make these layer representations more accessible and meaningful, we introduce a web-based tool that visualizes them both at the sentence and token level. We present three use cases. The first analyses gender issues in contextual word embeddings. The second and third are showing multilingual intermediate representations for sentences and tokens and the evolution of these intermediate representations along with the multiple layers of the decoder and in the context of multilingual machine translation.- Anthology ID:
- D19-3026
- Volume:
- Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations
- Month:
- November
- Year:
- 2019
- Address:
- Hong Kong, China
- Editors:
- Sebastian Padó, Ruihong Huang
- Venues:
- EMNLP | IJCNLP
- SIG:
- SIGDAT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 151–156
- Language:
- URL:
- https://aclanthology.org/D19-3026
- DOI:
- 10.18653/v1/D19-3026
- Cite (ACL):
- Carlos Escolano, Marta R. Costa-jussà, Elora Lacroux, and Pere-Pau Vázquez. 2019. Multilingual, Multi-scale and Multi-layer Visualization of Intermediate Representations. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): System Demonstrations, pages 151–156, Hong Kong, China. Association for Computational Linguistics.
- Cite (Informal):
- Multilingual, Multi-scale and Multi-layer Visualization of Intermediate Representations (Escolano et al., EMNLP-IJCNLP 2019)
- PDF:
- https://preview.aclanthology.org/ingest-bitext-workshop/D19-3026.pdf