@inproceedings{agarwal-etal-2026-image,
    title = "Do Image{--}Text Metrics Respect Semantic Invariances?",
    author = "Agarwal, Amit  and
      Patel, Hitesh Laxmichand  and
      Liu, Meizhu  and
      Singh, Jyotika  and
      Dua, Karan  and
      Meghwani, Hansa  and
      Rowe, Matthew  and
      Avendi, M.  and
      Abbasi, Yassi  and
      Sheng, Tao  and
      Ravi, Sujith  and
      Roth, Dan",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.findings-acl.1948/",
    pages = "39089--39116",
    ISBN = "979-8-89176-395-1",
    abstract = "Reference-free image{--}to{--}text evaluators are now standard for scoring image{--}caption alignment, yet it is unclear whether they respect semantic invariances. We present an invariance probe on five popular evaluators (CLIPScore, PAC-S, UMIC, FLEUR, and a deterministic LLM judge) under semantics-preserving perturbations along three axes: spatial (flips, context-preserving repositioning, light rotations), object (scale, category), and socio-linguistic framing (cultural/economic adjectives with neutral and length-matched controls). Across curated slices of three detection datasets and three caption evaluation suites, we find consistent non-semantic sensitivities: benign spatial edits and simple phrasing changes shift scores by ({\ensuremath{\approx}})6{--}9{\%} on average, and for systems separated by just 0.7{\%} these shifts can cause ranking flips in upto ({\ensuremath{\sim}})37{\%} of cases, particularly under spatial changes. A small human study also supports this finding and confirms that annotators generally judge perturbed pairs as equally correct, so these shifts reflect metric behavior rather than semantic change. We further propose invariance-calibrated scoring, a post-hoc adjustment that roughly halves median absolute sensitivity while retaining correlation with learned caption evaluators."
}