Abstract
Generative language models trained on large, diverse corpora can answer questions about a passage by generating the most likely continuation of the passage followed by a question/answer pair. However, accuracy rates vary depending on the type of question asked. In this paper we keep the passage fixed, and test with a wide variety of question types, exploring the strengths and weaknesses of the GPT-3 language model. We provide the passage and test questions as a challenge set for other language models.- Anthology ID:
- 2021.mrqa-1.7
- Volume:
- Proceedings of the 3rd Workshop on Machine Reading for Question Answering
- Month:
- November
- Year:
- 2021
- Address:
- Punta Cana, Dominican Republic
- Editors:
- Adam Fisch, Alon Talmor, Danqi Chen, Eunsol Choi, Minjoon Seo, Patrick Lewis, Robin Jia, Sewon Min
- Venue:
- MRQA
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 73–81
- Language:
- URL:
- https://aclanthology.org/2021.mrqa-1.7
- DOI:
- 10.18653/v1/2021.mrqa-1.7
- Cite (ACL):
- Douglas Summers-Stay, Claire Bonial, and Clare Voss. 2021. What Can a Generative Language Model Answer About a Passage?. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 73–81, Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- What Can a Generative Language Model Answer About a Passage? (Summers-Stay et al., MRQA 2021)
- PDF:
- https://preview.aclanthology.org/emnlp22-frontmatter/2021.mrqa-1.7.pdf
- Data
- GLUE