Artemis: A Novel Annotation Methodology for Indicative Single Document Summarization
Rahul Jha, Keping Bi, Yang Li, Mahdi Pakdaman, Asli Celikyilmaz, Ivan Zhiboedov, Kieran McDonald
Abstract
We describe Artemis (Annotation methodology for Rich, Tractable, Extractive, Multi-domain, Indicative Summarization), a novel hierarchical annotation process that produces indicative summaries for documents from multiple domains. Current summarization evaluation datasets are single-domain and focused on a few domains for which naturally occurring summaries can be easily found, such as news and scientific articles. These are not sufficient for training and evaluation of summarization models for use in document management and information retrieval systems, which need to deal with documents from multiple domains. Compared to other annotation methods such as Relative Utility and Pyramid, Artemis is more tractable because judges don’t need to look at all the sentences in a document when making an importance judgment for one of the sentences, while providing similarly rich sentence importance annotations. We describe the annotation process in detail and compare it with other similar evaluation systems. We also present analysis and experimental results over a sample set of 532 annotated documents.- Anthology ID:
- 2020.eval4nlp-1.8
- Volume:
- Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Venue:
- Eval4NLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 69–78
- Language:
- URL:
- https://aclanthology.org/2020.eval4nlp-1.8
- DOI:
- 10.18653/v1/2020.eval4nlp-1.8
- Cite (ACL):
- Rahul Jha, Keping Bi, Yang Li, Mahdi Pakdaman, Asli Celikyilmaz, Ivan Zhiboedov, and Kieran McDonald. 2020. Artemis: A Novel Annotation Methodology for Indicative Single Document Summarization. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 69–78, Online. Association for Computational Linguistics.
- Cite (Informal):
- Artemis: A Novel Annotation Methodology for Indicative Single Document Summarization (Jha et al., Eval4NLP 2020)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2020.eval4nlp-1.8.pdf
- Data
- CNN/Daily Mail