@inproceedings{rajaby-faghihi-etal-2020-latent,
    title = "Latent Alignment of Procedural Concepts in Multimodal Recipes",
    author = "Rajaby Faghihi, Hossein  and
      Mirzaee, Roshanak  and
      Paliwal, Sudarshan  and
      Kordjamshidi, Parisa",
    editor = "Wang, Xin  and
      Thomason, Jesse  and
      Hu, Ronghang  and
      Chen, Xinlei  and
      Anderson, Peter  and
      Wu, Qi  and
      Celikyilmaz, Asli  and
      Baldridge, Jason  and
      Wang, William Yang",
    booktitle = "Proceedings of the First Workshop on Advances in Language and Vision Research",
    month = jul,
    year = "2020",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest_wac_2008/2020.alvr-1.5/",
    doi = "10.18653/v1/2020.alvr-1.5",
    pages = "26--31",
    abstract = "We propose a novel alignment mechanism to deal with procedural reasoning on a newly released multimodal QA dataset, named RecipeQA. Our model is solving the textual cloze task which is a reading comprehension on a recipe containing images and instructions. We exploit the power of attention networks, cross-modal representations, and a latent alignment space between instructions and candidate answers to solve the problem. We introduce constrained max-pooling which refines the max pooling operation on the alignment matrix to impose disjoint constraints among the outputs of the model. Our evaluation result indicates a 19{\%} improvement over the baselines."
}