Fine-grained Multi-lingual Disentangled Autoencoder for Language-agnostic Representation Learning
Zetian Wu, Zhongkai Sun, Zhengyang Zhao, Sixing Lu, Chengyuan Ma, Chenlei Guo
Abstract
Encoding both language-specific and language-agnostic information into a single high-dimensional space is a common practice of pre-trained Multi-lingual Language Models (pMLM). Such encoding has been shown to perform effectively on natural language tasks requiring semantics of the whole sentence (e.g., translation). However, its effectiveness appears to be limited on tasks requiring partial information of the utterance (e.g., multi-lingual entity retrieval, template retrieval, and semantic alignment). In this work, a novel Fine-grained Multilingual Disentangled Autoencoder (FMDA) is proposed to disentangle fine-grained semantic information from language-specific information in a multi-lingual setting. FMDA is capable of successfully extracting the disentangled template semantic and residual semantic representations. Experiments conducted on the MASSIVE dataset demonstrate that the disentangled encoding can boost each other during the training, thus consistently outperforming the original pMLM and the strong language disentanglement baseline on monolingual template retrieval and cross-lingual semantic retrieval tasks across multiple languages.- Anthology ID:
- 2022.mmnlu-1.2
- Volume:
- Proceedings of the Massively Multilingual Natural Language Understanding Workshop (MMNLU-22)
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates (Hybrid)
- Editors:
- Jack FitzGerald, Kay Rottmann, Julia Hirschberg, Mohit Bansal, Anna Rumshisky, Charith Peris, Christopher Hench
- Venue:
- MMNLU
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 12–24
- Language:
- URL:
- https://aclanthology.org/2022.mmnlu-1.2
- DOI:
- 10.18653/v1/2022.mmnlu-1.2
- Cite (ACL):
- Zetian Wu, Zhongkai Sun, Zhengyang Zhao, Sixing Lu, Chengyuan Ma, and Chenlei Guo. 2022. Fine-grained Multi-lingual Disentangled Autoencoder for Language-agnostic Representation Learning. In Proceedings of the Massively Multilingual Natural Language Understanding Workshop (MMNLU-22), pages 12–24, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
- Cite (Informal):
- Fine-grained Multi-lingual Disentangled Autoencoder for Language-agnostic Representation Learning (Wu et al., MMNLU 2022)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2022.mmnlu-1.2.pdf