@inproceedings{moradi-etal-2019-interrogating,
    title = "Interrogating the Explanatory Power of Attention in Neural Machine Translation",
    author = "Moradi, Pooya  and
      Kambhatla, Nishant  and
      Sarkar, Anoop",
    editor = "Birch, Alexandra  and
      Finch, Andrew  and
      Hayashi, Hiroaki  and
      Konstas, Ioannis  and
      Luong, Thang  and
      Neubig, Graham  and
      Oda, Yusuke  and
      Sudoh, Katsuhito",
    booktitle = "Proceedings of the 3rd Workshop on Neural Generation and Translation",
    month = nov,
    year = "2019",
    address = "Hong Kong",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/D19-5624/",
    doi = "10.18653/v1/D19-5624",
    pages = "221--230",
    abstract = "Attention models have become a crucial component in neural machine translation (NMT). They are often implicitly or explicitly used to justify the model{'}s decision in generating a specific token but it has not yet been rigorously established to what extent attention is a reliable source of information in NMT. To evaluate the explanatory power of attention for NMT, we examine the possibility of yielding the same prediction but with counterfactual attention models that modify crucial aspects of the trained attention model. Using these counterfactual attention mechanisms we assess the extent to which they still preserve the generation of function and content words in the translation process. Compared to a state of the art attention model, our counterfactual attention models produce 68{\%} of function words and 21{\%} of content words in our German-English dataset. Our experiments demonstrate that attention models by themselves cannot reliably explain the decisions made by a NMT model."
}