@inproceedings{yu-etal-2025-evaluating,
    title = "Evaluating Multimodal Large Language Models on Video Captioning via {M}onte {C}arlo Tree Search",
    author = "Yu, Linhao  and
      Ji, Xingguang  and
      Liu, Yahui  and
      Kong, Fanheng  and
      Sun, Chenxi  and
      Zhang, Jingyuan  and
      Zhang, Hongzhi  and
      W., V.  and
      Zhang, Fuzheng  and
      Xiong, Deyi",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.323/",
    pages = "6435--6462",
    ISBN = "979-8-89176-251-0",
    abstract = "Video captioning can be used to assess the video understanding capabilities of Multimodal Large Language Models (MLLMs).However, existing benchmarks and evaluation protocols suffer from crucial issues, such as inadequate or homogeneous creation of key points, exorbitant cost of data creation, and limited evaluation scopes. To address these issues, we propose an automatic framework, named AutoCaption, which leverages Monte Carlo Tree Search (MCTS) to construct numerous and diverse descriptive sentences (\textit{i.e.}, key points) that thoroughly represent video content in an iterative way. This iterative captioning strategy enables the continuous enhancement of video details such as actions, objects' attributes, environment details, etc. We apply AutoCaption to curate MCTS-VCB, a fine-grained video caption benchmark covering video details, thereby enabling a comprehensive evaluation of MLLMs on the video captioning task. We evaluate more than 20 open- and closed-source MLLMs of varying sizes on MCTS-VCB. Results show that MCTS-VCB can effectively and comprehensively evaluate the video captioning capability, with Gemini-1.5-Pro achieving the highest F1 score of 71.2. Interestingly, we fine-tune InternVL2.5-8B with the AutoCaption-generated data, which helps the model achieve an overall improvement of 25.0{\%} on MCTS-VCB and 16.3{\%} on DREAM-1K, further demonstrating the effectiveness of AutoCaption. The code and data are available at \url{https://github.com/tjunlp-lab/MCTS-VCB}."
}