English to Hindi Multi-modal Neural Machine Translation and Hindi Image Captioning
Sahinur Rahman Laskar, Rohit Pratap Singh, Partha Pakray, Sivaji Bandyopadhyay
Abstract
With the widespread use of Machine Trans-lation (MT) techniques, attempt to minimizecommunication gap among people from di-verse linguistic backgrounds. We have par-ticipated in Workshop on Asian Transla-tion 2019 (WAT2019) multi-modal translationtask. There are three types of submissiontrack namely, multi-modal translation, Hindi-only image captioning and text-only transla-tion for English to Hindi translation. The mainchallenge is to provide a precise MT output. The multi-modal concept incorporates textualand visual features in the translation task. Inthis work, multi-modal translation track re-lies on pre-trained convolutional neural net-works (CNN) with Visual Geometry Grouphaving 19 layered (VGG19) to extract imagefeatures and attention-based Neural MachineTranslation (NMT) system for translation. The merge-model of recurrent neural network(RNN) and CNN is used for the Hindi-onlyimage captioning. The text-only translationtrack is based on the transformer model of theNMT system. The official results evaluated atWAT2019 translation task, which shows thatour multi-modal NMT system achieved Bilin-gual Evaluation Understudy (BLEU) score20.37, Rank-based Intuitive Bilingual Eval-uation Score (RIBES) 0.642838, Adequacy-Fluency Metrics (AMFM) score 0.668260 forchallenge test data and BLEU score 40.55,RIBES 0.760080, AMFM score 0.770860 forevaluation test data in English to Hindi multi-modal translation respectively.- Anthology ID:
- D19-5205
- Volume:
- Proceedings of the 6th Workshop on Asian Translation
- Month:
- November
- Year:
- 2019
- Address:
- Hong Kong, China
- Editors:
- Toshiaki Nakazawa, Chenchen Ding, Raj Dabre, Anoop Kunchukuttan, Nobushige Doi, Yusuke Oda, Ondřej Bojar, Shantipriya Parida, Isao Goto, Hidaya Mino
- Venue:
- WAT
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 62–67
- Language:
- URL:
- https://aclanthology.org/D19-5205
- DOI:
- 10.18653/v1/D19-5205
- Cite (ACL):
- Sahinur Rahman Laskar, Rohit Pratap Singh, Partha Pakray, and Sivaji Bandyopadhyay. 2019. English to Hindi Multi-modal Neural Machine Translation and Hindi Image Captioning. In Proceedings of the 6th Workshop on Asian Translation, pages 62–67, Hong Kong, China. Association for Computational Linguistics.
- Cite (Informal):
- English to Hindi Multi-modal Neural Machine Translation and Hindi Image Captioning (Laskar et al., WAT 2019)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/D19-5205.pdf