@inproceedings{schneider-etal-2021-towards,
    title = "Towards Multi-Modal Text-Image Retrieval to improve Human Reading",
    author = {Schneider, Florian  and
      Ala{\c{c}}am, {\"O}zge  and
      Wang, Xintong  and
      Biemann, Chris},
    editor = "Durmus, Esin  and
      Gupta, Vivek  and
      Liu, Nelson  and
      Peng, Nanyun  and
      Su, Yu",
    booktitle = "Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop",
    month = jun,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2021.naacl-srw.21/",
    abstract = "In primary school, children`s books, as well as in modern language learning apps, multi-modal learning strategies like illustrations of terms and phrases are used to support reading comprehension. Also, several studies in educational psychology suggest that integrating cross-modal information will improve reading comprehension. We claim that state-of- he-art multi-modal transformers, which could be used in a language learner context to improve human reading, will perform poorly because of the short and relatively simple textual data those models are trained with. To prove our hypotheses, we collected a new multi-modal image-retrieval dataset based on data from Wikipedia. In an in-depth data analysis, we highlight the differences between our dataset and other popular datasets. Additionally, we evaluate several state-of-the-art multi-modal transformers on text-image retrieval on our dataset and analyze their meager results, which verify our claims."
}