An Empirical Study of Adequate Vision Span for Attention-Based Neural Machine Translation

Raphael Shu, Hideki Nakayama


Abstract
Recently, the attention mechanism plays a key role to achieve high performance for Neural Machine Translation models. However, as it computes a score function for the encoder states in all positions at each decoding step, the attention model greatly increases the computational complexity. In this paper, we investigate the adequate vision span of attention models in the context of machine translation, by proposing a novel attention framework that is capable of reducing redundant score computation dynamically. The term “vision span”’ means a window of the encoder states considered by the attention model in one step. In our experiments, we found that the average window size of vision span can be reduced by over 50% with modest loss in accuracy on English-Japanese and German-English translation tasks.
Anthology ID:
W17-3201
Volume:
Proceedings of the First Workshop on Neural Machine Translation
Month:
August
Year:
2017
Address:
Vancouver
Venue:
NGT
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
1–10
Language:
URL:
https://aclanthology.org/W17-3201
DOI:
10.18653/v1/W17-3201
Bibkey:
Cite (ACL):
Raphael Shu and Hideki Nakayama. 2017. An Empirical Study of Adequate Vision Span for Attention-Based Neural Machine Translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 1–10, Vancouver. Association for Computational Linguistics.
Cite (Informal):
An Empirical Study of Adequate Vision Span for Attention-Based Neural Machine Translation (Shu & Nakayama, NGT 2017)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/W17-3201.pdf
Data
ASPEC