PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning

Yongil Kim, Yerin Hwang, Hyeongu Yun, Seunghyun Yoon, Trung Bui, Kyomin Jung

Abstract
Vulnerability to lexical perturbation is a critical weakness of automatic evaluation metrics for image captioning. This paper proposes Perturbation Robust Multi-Lingual CLIPScore(PR-MCS), which exhibits robustness to such perturbations, as a novel reference-free image captioning metric applicable to multiple languages. To achieve perturbation robustness, we fine-tune the text encoder of CLIP with our language-agnostic method to distinguish the perturbed text from the original text. To verify the robustness of PR-MCS, we introduce a new fine-grained evaluation dataset consisting of detailed captions, critical objects, and the relationships between the objects for 3,000 images in five languages. In our experiments, PR-MCS significantly outperforms baseline metrics in capturing lexical noise of all various perturbation types in all five languages, while maintaining a strong correlation with human judgments.
Anthology ID:
2023.findings-emnlp.819
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
12237–12258
Language:
URL:
https://aclanthology.org/2023.findings-emnlp.819
DOI:
10.18653/v1/2023.findings-emnlp.819
Bibkey:
Cite (ACL):
Yongil Kim, Yerin Hwang, Hyeongu Yun, Seunghyun Yoon, Trung Bui, and Kyomin Jung. 2023. PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12237–12258, Singapore. Association for Computational Linguistics.
Cite (Informal):
PR-MCS: Perturbation Robust Metric for MultiLingual Image Captioning (Kim et al., Findings 2023)
Copy Citation:
PDF:
https://preview.aclanthology.org/naacl24-info/2023.findings-emnlp.819.pdf
PDF Search