LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation

Hongcheng Guo, Jiaheng Liu, Haoyang Huang, Jian Yang, Zhoujun Li, Dongdong Zhang, Zheng Cui

Abstract
Multimodal Machine Translation (MMT) focuses on enhancing text-only translation with visual features, which has attracted considerable attention from both natural language processing and computer vision communities. Recent advances still struggle to train a separate model for each language pair, which is costly and unaffordable when the number of languages increases in the real world. In other words, the multilingual multimodal machine translation (Multilingual MMT) task has not been investigated, which aims to handle the aforementioned issues by providing a shared semantic space for multiple languages. Besides, the image modality has no language boundaries, which is superior to bridging the semantic gap between languages. To this end,we first propose the Multilingual MMT task by establishing two new Multilingual MMT benchmark datasets covering seven languages.Then, an effective baseline LVP-M3 using visual prompts is proposed to support translations between different languages,which includes three stages (token encoding, language-aware visual prompt generation, and language translation). Extensive experimental results on our constructed benchmark datasets demonstrate the effectiveness of LVP-M3 method for Multilingual MMT.
Anthology ID:
2022.emnlp-main.184
Volume:
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2862–2872
Language:
URL:
https://aclanthology.org/2022.emnlp-main.184
DOI:
10.18653/v1/2022.emnlp-main.184
Bibkey:
Cite (ACL):
Hongcheng Guo, Jiaheng Liu, Haoyang Huang, Jian Yang, Zhoujun Li, Dongdong Zhang, and Zheng Cui. 2022. LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2862–2872, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
LVP-M3: Language-aware Visual Prompt for Multilingual Multimodal Machine Translation (Guo et al., EMNLP 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl24-info/2022.emnlp-main.184.pdf
PDF Search