Enhancing Video Translation Context with Object Labels
Jeremy Gwinnup, Tim Anderson, Brian Ore, Eric Hansen, Kevin Duh
Abstract
We present a simple yet efficient method to enhance the quality of machine translation models trained on multimodal corpora by augmenting the training text with labels of detected objects in the corresponding video segments. We then test the effects of label augmentation in both baseline and two automatic speech recognition (ASR) conditions. In contrast with multimodal techniques that merge visual and textual features, our modular method is easy to implement and the results are more interpretable. Comparisons are made with Transformer translation architectures trained with baseline and augmented labels, showing improvements of up to +1.0 BLEU on the How2 dataset.- Anthology ID:
- 2023.iwslt-1.8
- Volume:
- Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada (in-person and online)
- Editors:
- Elizabeth Salesky, Marcello Federico, Marine Carpuat
- Venue:
- IWSLT
- SIG:
- SIGSLT
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 130–137
- Language:
- URL:
- https://aclanthology.org/2023.iwslt-1.8
- DOI:
- 10.18653/v1/2023.iwslt-1.8
- Cite (ACL):
- Jeremy Gwinnup, Tim Anderson, Brian Ore, Eric Hansen, and Kevin Duh. 2023. Enhancing Video Translation Context with Object Labels. In Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 130–137, Toronto, Canada (in-person and online). Association for Computational Linguistics.
- Cite (Informal):
- Enhancing Video Translation Context with Object Labels (Gwinnup et al., IWSLT 2023)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2023.iwslt-1.8.pdf