Abstract
Recently it has been shown that without any access to the contextual passage, multiple choice reading comprehension (MCRC) systems are able to answer questions significantly better than random on average. These systems use their accumulated “world knowledge” to directly answer questions, rather than using information from the passage. This paper examines the possibility of exploiting this observation as a tool for test designers to ensure that the form of “world knowledge” is acceptable for a particular set of questions. We propose information-theory based metrics that enable the level of “world knowledge” exploited by systems to be assessed. Two metrics are described: the expected number of options, which measures whether a passage-free system can identify the answer a question using world knowledge; and the contextual mutual information, which measures the importance of context for a given question. We demonstrate that questions with low expected number of options, and hence answerable by the shortcut system, are often similarly answerable by humans without context. This highlights that the general knowledge ‘shortcuts’ could be equally used by exam candidates, and that our proposed metrics may be helpful for future test designers to monitor the quality of questions.- Anthology ID:
- 2023.fever-1.5
- Volume:
- Proceedings of the Sixth Fact Extraction and VERification Workshop (FEVER)
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Mubashara Akhtar, Rami Aly, Christos Christodoulopoulos, Oana Cocarascu, Zhijiang Guo, Arpit Mittal, Michael Schlichtkrull, James Thorne, Andreas Vlachos
- Venue:
- FEVER
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 49–57
- Language:
- URL:
- https://aclanthology.org/2023.fever-1.5
- DOI:
- 10.18653/v1/2023.fever-1.5
- Cite (ACL):
- Adian Liusie, Vatsal Raina, and Mark Gales. 2023. “World Knowledge” in Multiple Choice Reading Comprehension. In Proceedings of the Sixth Fact Extraction and VERification Workshop (FEVER), pages 49–57, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- “World Knowledge” in Multiple Choice Reading Comprehension (Liusie et al., FEVER 2023)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2023.fever-1.5.pdf