@inproceedings{charles-etal-2025-diagnostic,
    title = "A Diagnostic Framework for Auditing Reference-Free Vision-Language Metrics",
    author = "Charles, Angeline  and
      Panda, Srikant  and
      Agarwal, Amit  and
      Patel, Hitesh Laxmichand  and
      Pattnayak, Priyaranjan  and
      Kumar, Bhargava  and
      Kumar, Tejaswini",
    editor = "Inui, Kentaro  and
      Sakti, Sakriani  and
      Wang, Haofen  and
      Wong, Derek F.  and
      Bhattacharyya, Pushpak  and
      Banerjee, Biplab  and
      Ekbal, Asif  and
      Chakraborty, Tanmoy  and
      Singh, Dhirendra Pratap",
    booktitle = "Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics",
    month = dec,
    year = "2025",
    address = "Mumbai, India",
    publisher = "The Asian Federation of Natural Language Processing and The Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-ijcnlp-aacl/2025.ijcnlp-long.142/",
    pages = "2633--2644",
    ISBN = "979-8-89176-298-5",
    abstract = "Reference-free metrics such as CLIPScore and PAC-S are increasingly used in vision-language tasks due to their scalability and independence from human-written references. However, their reliability under linguistic, visual, and cultural variation remains underexplored. In this work, we present a systematic audit of CLIPScore and PAC-S using an eight-factor diagnostic framework applied to MS-COCO validation images. Our analysis reveals consistent failure modes across dimensions including object size, content category, syntax, named entities, spatial relations and cultural context. Both metrics penalize captions referencing African ({\ensuremath{-}}5.5{\%}, {\ensuremath{-}}4.8{\%}) and Arabian ({\ensuremath{-}}4.9{\%}, {\ensuremath{-}}5.3{\%}) cultures, favor large-object and animal-centric scenes (by 20-30{\%}) and show limited sensitivity to spatial negation and word order. CLIPScore correlates more strongly with syntactic complexity, while PAC-S demonstrates greater robustness to verbosity and named{--}entity variation highlighting complementary strengths rather than superiority. These findings expose cultural and content bias, weak semantic robustness, and limited compositional understanding. We conclude with design recommendations to improve fairness, scale invariance, and semantic grounding in future reference-free evaluation metrics."
}