Chuang Han
2025
SHIFT: Selected Helpful Informative Frame for Video-guided Machine Translation
Boyu Guan
|
Chuang Han
|
Yining Zhang
|
Yupu Liang
|
Zhiyang Zhang
|
Yang Zhao
|
Chengqing Zong
Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing
Video-guided Machine Translation (VMT) aims to improve translation quality by integrating contextual information from paired short video clips. Mainstream VMT approaches typically incorporate multimodal information by uniformly sampling frames from the input videos. However, this paradigm frequently incurs significant computational overhead and introduces redundant multimodal content, which degrades both efficiency and translation quality. To tackle these challenges, we propose SHIFT (Selected Helpful Informative Frame for Translation). It is a lightweight, plug-and-play framework designed for VMT with Multimodal Large Language Models (MLLMs). SHIFT adaptively selects a single informative key frame when visual context is necessary; otherwise, it relies solely on textual input. This process is guided by a dedicated clustering module and a selector module. Experimental results demonstrate that SHIFT enhances the performance of MLLMs on the VMT task while simultaneously reducing computational cost, without sacrificing generalization ability. The code will be released upon acceptance.
Search
Fix author
Co-authors
- Boyu Guan 1
- Yupu Liang 1
- Yining Zhang 1
- Zhiyang Zhang 1
- Yang Zhao 1
- show all...