Abstract
So far work on automatic summarization has dealt primarily with English data. Accordingly, evaluation methods were primarily developed with this language in mind. In our work, we present experiments of adapting available evaluation methods such as ROUGE and PYRAMID to non-English data. We base our experiments on various English and non-English homogeneous benchmark data sets as well as a non-English heterogeneous data set. Our results indicate that ROUGE can indeed be adapted to non-English data – both homogeneous and heterogeneous. Using a recent implementation of performing an automatic PYRAMID evaluation, we also show its adaptability to non-English data.- Anthology ID:
- 2020.lrec-1.822
- Volume:
- Proceedings of the Twelfth Language Resources and Evaluation Conference
- Month:
- May
- Year:
- 2020
- Address:
- Marseille, France
- Editors:
- Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis
- Venue:
- LREC
- SIG:
- Publisher:
- European Language Resources Association
- Note:
- Pages:
- 6656–6662
- Language:
- English
- URL:
- https://aclanthology.org/2020.lrec-1.822
- DOI:
- Cite (ACL):
- Christopher Tauchmann and Margot Mieskes. 2020. Language Agnostic Automatic Summarization Evaluation. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6656–6662, Marseille, France. European Language Resources Association.
- Cite (Informal):
- Language Agnostic Automatic Summarization Evaluation (Tauchmann & Mieskes, LREC 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-2/2020.lrec-1.822.pdf