@inproceedings{nguyen-etal-2025-memeqa,
    title = "{M}eme{QA}: Holistic Evaluation for Meme Understanding",
    author = "Nguyen, Khoi P. N.  and
      Li, Terrence  and
      Zhou, Derek Lou  and
      Xiong, Gabriel  and
      Balu, Pranav  and
      Alahari, Nandhan  and
      Huang, Alan  and
      Chauhan, Tanush  and
      Bala, Harshavardhan  and
      Guzelordu, Emre  and
      Kashfi, Affan  and
      Xu, Aaron  and
      Shrestha, Suyesh  and
      Vu, Megan  and
      Wang, Jerry  and
      Ng, Vincent",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.927/",
    pages = "18926--18946",
    ISBN = "979-8-89176-251-0",
    abstract = "Automated meme understanding requires systems to demonstrate fine-grained visual recognition, commonsense reasoning, and extensive cultural knowledge. However, existing benchmarks for meme understanding only concern narrow aspects of meme semantics. To fill this gap, we present MemeQA, a dataset of over 9,000 multiple-choice questions designed to holistically evaluate meme comprehension across seven cognitive aspects. Experiments show that state-of-the-art Large Multimodal Models perform much worse than humans on MemeQA. While fine-tuning improves their performance, they still make many errors on memes wherein proper understanding requires going beyond surface-level sentiment. Moreover, injecting ``None of the above'' into the available options makes the questions more challenging for the models. Our dataset is publicly available at https://github.com/npnkhoi/memeqa."
}