@inproceedings{li-etal-2021-feature,
    title = "Feature-level Incongruence Reduction for Multimodal Translation",
    author = "Li, Zhifeng  and
      Hong, Yu  and
      Pan, Yuchen  and
      Tang, Jian  and
      Yao, Jianmin  and
      Zhou, Guodong",
    editor = "{Xin}  and
      Hu, Ronghang  and
      Hudson, Drew  and
      Fu, Tsu-Jui  and
      Rohrbach, Marcus  and
      Fried, Daniel",
    booktitle = "Proceedings of the Second Workshop on Advances in Language and Vision Research",
    month = jun,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2021.alvr-1.1",
    doi = "10.18653/v1/2021.alvr-1.1",
    pages = "1--10",
    abstract = "Caption translation aims to translate image annotations (captions for short). Recently, Multimodal Neural Machine Translation (MNMT) has been explored as the essential solution. Besides of linguistic features in captions, MNMT allows visual(image) features to be used. The integration of multimodal features reinforces the semantic representation and considerably improves translation performance. However, MNMT suffers from the incongruence between visual and linguistic features. To overcome the problem, we propose to extend MNMT architecture with a harmonization network, which harmonizes multimodal features(linguistic and visual features)by unidirectional modal space conversion. It enables multimodal translation to be carried out in a seemingly monomodal translation pipeline. We experiment on the golden Multi30k-16 and 17. Experimental results show that, compared to the baseline,the proposed method yields the improvements of 2.2{\%} BLEU for the scenario of translating English captions into German (En→De) at best,7.6{\%} for the case of English-to-French translation(En→Fr) and 1.5{\%} for English-to-Czech(En→Cz). The utilization of harmonization network leads to the competitive performance to the-state-of-the-art.",
}