@inproceedings{wu-etal-2025-realhitbench,
    title = "{R}eal{H}i{TB}ench: A Comprehensive Realistic Hierarchical Table Benchmark for Evaluating {LLM}-Based Table Analysis",
    author = "Wu, Pengzuo  and
      Yang, Yuhang  and
      Zhu, Guangcheng  and
      Ye, Chao  and
      Gu, Hong  and
      Lu, Xu  and
      Xiao, Ruixuan  and
      Bao, Bowen  and
      He, Yijing  and
      Zha, Liangyu  and
      Ye, Wentao  and
      Zhao, Junbo  and
      Wang, Haobo",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.371/",
    pages = "7105--7137",
    ISBN = "979-8-89176-256-5",
    abstract = "With the rapid advancement of Large Language Models (LLMs), there is an increasing need for challenging benchmarks to evaluate their capabilities in handling complex tabular data. However, existing benchmarks are either based on outdated data setups or focus solely on simple, flat table structures. In this paper, we introduce **RealHiTBench**, a comprehensive benchmark designed to evaluate the performance of both LLMs and Multimodal LLMs (MLLMs) across a variety of input formats for complex tabular data, including LaTeX, HTML, and PNG. RealHiTBench also includes a diverse collection of tables with intricate structures, spanning a wide range of task types. Our experimental results, using **25** state-of-the-art LLMs, demonstrate that RealHiTBench is indeed a challenging benchmark. Moreover, we also develop TreeThinker, a tree-based agent that organizes hierarchical headers into a tree structure for enhanced tabular reasoning, validating the importance of improving LLMs' perception of table hierarchies. We hope that our work will inspire further research on tabular data reasoning and the development of more robust models. The code and data are available at https://github.com/cspzyy/RealHiTBench."
}