@inproceedings{masry-etal-2025-chartqapro,
    title = "{C}hart{QAP}ro: A More Diverse and Challenging Benchmark for Chart Question Answering",
    author = "Masry, Ahmed  and
      Islam, Mohammed Saidul  and
      Ahmed, Mahir  and
      Bajaj, Aayush  and
      Kabir, Firoz  and
      Kartha, Aaryaman  and
      Laskar, Md Tahmid Rahman  and
      Rahman, Mizanur  and
      Rahman, Shadikur  and
      Shahmohammadi, Mehrad  and
      Thakkar, Megh  and
      Parvez, Md Rizwan  and
      Hoque, Enamul  and
      Joty, Shafiq",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.findings-acl.978/",
    pages = "19123--19151",
    ISBN = "979-8-89176-256-5",
    abstract = "Charts are ubiquitous, as people often use them to analyze data, answer questions, and discover critical insights. However, performing complex analytical tasks with charts requires significant perceptual and cognitive effort. Chart Question Answering (CQA) systems automate this process by enabling models to interpret and reason with visual representations of data. However, existing benchmarks like ChartQA lack real-world diversity and have recently shown performance saturation with modern large vision-language models (LVLMs). To address these limitations, we introduce ChartQAPro, a new benchmark that includes 1,341 charts from 99 diverse sources, spanning various chart types{---}including infographics and dashboards{---}and featuring 1,948 questions in various types, such as multiple-choice, conversational, hypothetical, and unanswerable questions, to better reflect real-world challenges. Our evaluations with 21 models show a substantial performance drop for LVLMs on ChartQAPro; e.g., Claude Sonnet 3.5 scores 90.5{\%} on ChartQA but only 55.81{\%} on ChartQAPro, underscoring the complexity of chart reasoning. We complement our findings with detailed error analyses and ablation studies, identifying key challenges and opportunities for advancing LVLMs in chart understanding and reasoning. We release ChartQAPro at https://github.com/vis-nlp/ChartQAPro."
}