Bend but Don’t Break? Multi-Challenge Stress Test for QA Models
Hemant Pugaliya, James Route, Kaixin Ma, Yixuan Geng, Eric Nyberg
Abstract
The field of question answering (QA) has seen rapid growth in new tasks and modeling approaches in recent years. Large scale datasets and focus on challenging linguistic phenomena have driven development in neural models, some of which have achieved parity with human performance in limited cases. However, an examination of state-of-the-art model output reveals that a gap remains in reasoning ability compared to a human, and performance tends to degrade when models are exposed to less-constrained tasks. We are interested in more clearly defining the strengths and limitations of leading models across diverse QA challenges, intending to help future researchers with identifying pathways to generalizable performance. We conduct extensive qualitative and quantitative analyses on the results of four models across four datasets and relate common errors to model capabilities. We also illustrate limitations in the datasets we examine and discuss a way forward for achieving generalizable models and datasets that broadly test QA capabilities.- Anthology ID:
- D19-5818
- Volume:
- Proceedings of the 2nd Workshop on Machine Reading for Question Answering
- Month:
- November
- Year:
- 2019
- Address:
- Hong Kong, China
- Editors:
- Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, Danqi Chen
- Venue:
- WS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 125–136
- Language:
- URL:
- https://aclanthology.org/D19-5818
- DOI:
- 10.18653/v1/D19-5818
- Cite (ACL):
- Hemant Pugaliya, James Route, Kaixin Ma, Yixuan Geng, and Eric Nyberg. 2019. Bend but Don’t Break? Multi-Challenge Stress Test for QA Models. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 125–136, Hong Kong, China. Association for Computational Linguistics.
- Cite (Informal):
- Bend but Don’t Break? Multi-Challenge Stress Test for QA Models (Pugaliya et al., 2019)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/D19-5818.pdf