Rethinking Efficient Multilingual Text Summarization Meta-Evaluation

Rilyn Han, Jiawen Chen, Yixin Liu, Arman Cohan

Abstract
Evaluating multilingual summarization evaluation metrics, i.e., meta-evaluation, is challenging because of the difficulty of human annotation collection. Therefore, we investigate an efficient multilingual meta-evaluation framework that uses machine translation systems to transform a monolingual meta-evaluation dataset into multilingual versions. To this end, we introduce a statistical test to verify the transformed dataset quality by checking the meta-evaluation result consistency on the original dataset and back-translated dataset. With this quality verification method, we transform an existing English summarization meta-evaluation dataset, RoSE, into 30 languages, and conduct a multilingual meta-evaluation of several representative automatic evaluation metrics. In our meta-evaluation, we find that metric performance varies in different languages and neural metrics generally outperform classical text-matching-based metrics in non-English languages. Moreover, we identify a two-stage evaluation method with superior performance, which first translates multilingual texts into English and then performs evaluation. We make the transformed datasets publicly available to facilitate future research.
Anthology ID:
2024.findings-acl.930
Volume:
Findings of the Association for Computational Linguistics ACL 2024
Month:
August
Year:
2024
Address:
Bangkok, Thailand and virtual meeting
Editors:
Lun-Wei Ku, Andre Martins, Vivek Srikumar
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
15739–15746
Language:
URL:
https://aclanthology.org/2024.findings-acl.930
DOI:
10.18653/v1/2024.findings-acl.930
Bibkey:
Cite (ACL):
Rilyn Han, Jiawen Chen, Yixin Liu, and Arman Cohan. 2024. Rethinking Efficient Multilingual Text Summarization Meta-Evaluation. In Findings of the Association for Computational Linguistics ACL 2024, pages 15739–15746, Bangkok, Thailand and virtual meeting. Association for Computational Linguistics.
Cite (Informal):
Rethinking Efficient Multilingual Text Summarization Meta-Evaluation (Han et al., Findings 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-5/2024.findings-acl.930.pdf
PDF Search