M3: Multi-level dataset for Multi-document summarisation of Medical studies
Yulia Otmakhova, Karin Verspoor, Timothy Baldwin, Antonio Jimeno Yepes, Jey Han Lau
Abstract
We present M3 (Multi-level dataset for Multi-document summarisation of Medical studies), a benchmark dataset for evaluating the quality of summarisation systems in the biomedical domain. The dataset contains sets of multiple input documents and target summaries of three levels of complexity: documents, sentences, and propositions. The dataset also includes several levels of annotation, including biomedical entities, direction, and strength of relations between them, and the discourse relationships between the input documents (“contradiction” or “agreement”). We showcase usage scenarios of the dataset by testing 10 generic and domain-specific summarisation models in a zero-shot setting, and introduce a probing task based on counterfactuals to test if models are aware of the direction and strength of the conclusions generated from input studies.- Anthology ID:
- 2022.findings-emnlp.286
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2022
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 3887–3901
- Language:
- URL:
- https://aclanthology.org/2022.findings-emnlp.286
- DOI:
- 10.18653/v1/2022.findings-emnlp.286
- Cite (ACL):
- Yulia Otmakhova, Karin Verspoor, Timothy Baldwin, Antonio Jimeno Yepes, and Jey Han Lau. 2022. M3: Multi-level dataset for Multi-document summarisation of Medical studies. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3887–3901, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- M3: Multi-level dataset for Multi-document summarisation of Medical studies (Otmakhova et al., Findings 2022)
- PDF:
- https://preview.aclanthology.org/proper-vol2-ingestion/2022.findings-emnlp.286.pdf