GANDALF: a General Character Name Description Dataset for Long Fiction
Fredrik Carlsson, Magnus Sahlgren, Fredrik Olsson, Amaru Cuba Gyllensten
Abstract
This paper introduces a long-range multiple-choice Question Answering (QA) dataset, based on full-length fiction book texts. The questions are formulated as 10-way multiple-choice questions, where the task is to select the correct character name given a character description, or vice-versa. Each character description is formulated in natural text and often contains information from several sections throughout the book. We provide 20,000 questions created from 10,000 manually annotated descriptions of characters from 177 books containing 152,917 words on average. We address the current discourse regarding dataset bias and leakage by a simple anonymization procedure, which in turn enables interesting probing possibilities. Finally, we show that suitable baseline algorithms perform very poorly on this task, with the book size itself making it non-trivial to attempt a Transformer-based QA solution. This leaves ample room for future improvement, and hints at the need for a completely different type of solution.- Anthology ID:
- 2021.mrqa-1.13
- Volume:
- Proceedings of the 3rd Workshop on Machine Reading for Question Answering
- Month:
- November
- Year:
- 2021
- Address:
- Punta Cana, Dominican Republic
- Editors:
- Adam Fisch, Alon Talmor, Danqi Chen, Eunsol Choi, Minjoon Seo, Patrick Lewis, Robin Jia, Sewon Min
- Venue:
- MRQA
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 119–132
- Language:
- URL:
- https://aclanthology.org/2021.mrqa-1.13
- DOI:
- 10.18653/v1/2021.mrqa-1.13
- Cite (ACL):
- Fredrik Carlsson, Magnus Sahlgren, Fredrik Olsson, and Amaru Cuba Gyllensten. 2021. GANDALF: a General Character Name Description Dataset for Long Fiction. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 119–132, Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- GANDALF: a General Character Name Description Dataset for Long Fiction (Carlsson et al., MRQA 2021)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2021.mrqa-1.13.pdf
- Data
- BookTest, GLUE, SuperGLUE