Abstract
In this paper, we employ Singular Value Canonical Correlation Analysis (SVCCA) to analyze representations learnt in a multilingual end-to-end speech translation model trained over 22 languages. SVCCA enables us to estimate representational similarity across languages and layers, enhancing our understanding of the functionality of multilingual speech translation and its potential connection to multilingual neural machine translation. The multilingual speech translation model is trained on the CoVoST 2 dataset in all possible directions, and we utilize LASER to extract parallel bitext data for SVCCA analysis. We derive three major findings from our analysis: (I) Linguistic similarity loses its efficacy in multilingual speech translation when the training data for a specific language is limited. (II) Enhanced encoder representations and well-aligned audio-text data significantly improve translation quality, surpassing the bilingual counterparts when the training data is not compromised. (III) The encoder representations of multilingual speech translation demonstrate superior performance in predicting phonetic features in linguistic typology prediction. With these findings, we propose that releasing the constraint of limited data for low-resource languages and subsequently combining them with linguistically related high-resource languages could offer a more effective approach for multilingual end-to-end speech translation.- Anthology ID:
- 2023.findings-emnlp.956
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 14332–14348
- Language:
- URL:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2023.findings-emnlp.956/
- DOI:
- 10.18653/v1/2023.findings-emnlp.956
- Cite (ACL):
- Haoran Sun, Xiaohu Zhao, Yikun Lei, Shaolin Zhu, and Deyi Xiong. 2023. Towards a Deep Understanding of Multilingual End-to-End Speech Translation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14332–14348, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Towards a Deep Understanding of Multilingual End-to-End Speech Translation (Sun et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/build-pipeline-with-new-library/2023.findings-emnlp.956.pdf