@inproceedings{cheng-etal-2025-caparena,
    title = "{C}ap{A}rena: Benchmarking and Analyzing Detailed Image Captioning in the {LLM} Era",
    author = "Cheng, Kanzhi  and
      Song, Wenpo  and
      Fan, Jiaxin  and
      Ma, Zheng  and
      Sun, Qiushi  and
      Xu, Fangzhi  and
      Yan, Chenyang  and
      Chen, Nuo  and
      Zhang, Jianbing  and
      Chen, Jiajun",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.findings-acl.724/",
    pages = "14077--14094",
    ISBN = "979-8-89176-256-5",
    abstract = "Image captioning has been a longstanding challenge in vision-language research. With the rise of LLMs, modern Vision-Language Models (VLMs) generate detailed and comprehensive image descriptions. However, benchmarking the quality of such captions remains unresolved. This paper addresses two key questions: (1) How well do VLMs actually perform on image captioning, particularly compared to humans? We built CapArena, a platform with over 6000 pairwise caption battles and high-quality human preference votes. Our Arena-style evaluation marks a milestone, showing that leading models like GPT-4o achieve or even surpass human performance, while most open-source models lag behind. (2) Can automated metrics reliably assess caption quality? Using human annotations from CapArena, we evaluate traditional and recent captioning metrics, as well as VLM-as-a-Judge. Our analysis reveals that while some metrics (e.g., METEOR) show high caption-level agreement with humans, their systematic biases lead to inconsistencies in model ranking. In contrast, VLM-as-a-Judge demonstrates robust discernment at both the caption and model levels. Building on these insights, we release CapArena-Auto, an accurate and efficient automated benchmark for detailed captioning, achieving 93.4{\%} correlation with human rankings at just {\$}4 per test. All data and evaluation resources have been open-sourced."
}