@inproceedings{hirota-etal-2025-lotus,
    title = "{LOTUS}: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences",
    author = "Hirota, Yusuke  and
      Li, Boyi  and
      Hachiuma, Ryo  and
      Wu, Yueh-Hua  and
      Ivanovic, Boris  and
      Pavone, Marco  and
      Choi, Yejin  and
      Wang, Yu-Chiang Frank  and
      Nakashima, Yuta  and
      Yang, Chao-Han Huck",
    editor = "Rehm, Georg  and
      Li, Yunyao",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.acl-industry.22/",
    pages = "295--309",
    ISBN = "979-8-89176-288-6",
    abstract = "Large Vision-Language Models (LVLMs) have transformed image captioning, shifting from concise captions to detailed descriptions. We introduce LOTUS, a leaderboard for evaluating detailed captions, addressing three main gaps in existing evaluations: lack of standardized criteria, bias-aware assessments, and user preference considerations. LOTUS comprehensively evaluates various aspects, including caption quality (e.g., alignment, descriptiveness), risks (e.g., hallucination), and societal biases (e.g., gender bias) while enabling preference-oriented evaluations by tailoring criteria to diverse user preferences. Our analysis of recent LVLMs reveals no single model excels across all criteria, while correlations emerge between caption detail and bias risks. Preference-oriented evaluations demonstrate that optimal model selection depends on user priorities."
}