@inproceedings{luo-etal-2025-finmme,
    title = "{F}in{MME}: Benchmark Dataset for Financial Multi-Modal Reasoning Evaluation",
    author = "Luo, Junyu  and
      Kou, Zhizhuo  and
      Yang, Liming  and
      Luo, Xiao  and
      Huang, Jinsheng  and
      Xiao, Zhiping  and
      Peng, Jingshu  and
      Liu, Chengzhong  and
      Ji, Jiaming  and
      Liu, Xuanzhe  and
      Han, Sirui  and
      Zhang, Ming  and
      Guo, Yike",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1426/",
    pages = "29465--29489",
    ISBN = "979-8-89176-251-0",
    abstract = "Multimodal Large Language Models (MLLMs) have experienced rapid development in recent years. However, in the financial domain, there is a notable lack of effective and specialized multimodal evaluation datasets. To advance the development of MLLMs in the finance domain, we introduce FinMME, encompassing more than 11,000 high-quality financial research samples across 18 financial domains and 6 asset classes, featuring 10 major chart types and 21 subtypes. We ensure data quality through 20 annotators and carefully designed validation mechanisms. Additionally, we develop FinScore, an evaluation system incorporating hallucination penalties and multi-dimensional capability assessment to provide an unbiased evaluation. Extensive experimental results demonstrate that even state-of-the-art models like GPT-4o exhibit unsatisfactory performance on FinMME, highlighting its challenging nature. The benchmark exhibits high robustness with prediction variations under different prompts remaining below 1{\%}, demonstrating superior reliability compared to existing datasets. Our dataset and evaluation protocol are available at https://huggingface.co/datasets/luojunyu/FinMME and https://github.com/luo-junyu/FinMME."
}