@inproceedings{wu-etal-2024-chartinsights,
    title = "{C}hart{I}nsights: Evaluating Multimodal Large Language Models for Low-Level Chart Question Answering",
    author = "Wu, Yifan  and
      Yan, Lutao  and
      Shen, Leixian  and
      Wang, Yunhai  and
      Tang, Nan  and
      Luo, Yuyu",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.findings-emnlp.710/",
    doi = "10.18653/v1/2024.findings-emnlp.710",
    pages = "12174--12200",
    abstract = "Chart question answering (ChartQA) tasks play a critical role in interpreting and extracting insights from visualization charts. While recent advancements in multimodal large language models (MLLMs) like GPT-4o have shown promise in high-level ChartQA tasks, such as chart captioning, their effectiveness in low-level ChartQA tasks (*e.g.*, identifying correlations) remains underexplored.In this paper, we address this gap by evaluating MLLMs on low-level ChartQA using a newly curated dataset, *ChartInsights*, which consists of 22,347 (chart, task, query, answer) covering 10 data analysis tasks across 7 chart types. We systematically evaluate 19 advanced MLLMs, including 12 open-source and 7 closed-source models. The average accuracy rate across these models is 39.8{\%}, with GPT-4o achieving the highest accuracy at 69.17{\%}.To further explore the limitations of MLLMs in low-level ChartQA, we conduct experiments that alter visual elements of charts (*e.g.*, changing color schemes, adding image noise) to assess their impact on the task effectiveness. Furthermore, we propose a new textual prompt strategy, *Chain-of-Charts*, tailored for low-level ChartQA tasks, which boosts performance by 14.41{\%}, achieving an accuracy of 83.58{\%}. Finally, incorporating a visual prompt strategy that directs attention to relevant visual elements further improves accuracy to 84.32{\%}."
}