@inproceedings{lala-etal-2019-grounded,
    title = "Grounded Word Sense Translation",
    author = "Lala, Chiraag  and
      Madhyastha, Pranava  and
      Specia, Lucia",
    editor = "Bernardi, Raffaella  and
      Fernandez, Raquel  and
      Gella, Spandana  and
      Kafle, Kushal  and
      Kanan, Christopher  and
      Lee, Stefan  and
      Nabi, Moin",
    booktitle = "Proceedings of the Second Workshop on Shortcomings in Vision and Language",
    month = jun,
    year = "2019",
    address = "Minneapolis, Minnesota",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/iwcs-25-ingestion/W19-1808/",
    doi = "10.18653/v1/W19-1808",
    pages = "78--85",
    abstract = "Recent work on visually grounded language learning has focused on broader applications of grounded representations, such as visual question answering and multimodal machine translation. In this paper we consider grounded word sense translation, i.e. the task of correctly translating an ambiguous source word given the corresponding textual and visual context. Our main objective is to investigate the extent to which images help improve word-level (lexical) translation quality. We do so by first studying the dataset for this task to understand the scope and challenges of the task. We then explore different data settings, image features, and ways of grounding to investigate the gain from using images in each of the combinations. We find that grounding on the image is specially beneficial in weaker unidirectional recurrent translation models. We observe that adding structured image information leads to stronger gains in lexical translation accuracy."
}