Have You Seen That Number? Investigating Extrapolation in Question Answering Models
Jeonghwan Kim, Giwon Hong, Kyung-min Kim, Junmo Kang, Sung-Hyon Myaeng
Abstract
Numerical reasoning in machine reading comprehension (MRC) has shown drastic improvements over the past few years. While the previous models for numerical MRC are able to interpolate the learned numerical reasoning capabilities, it is not clear whether they can perform just as well on numbers unseen in the training dataset. Our work rigorously tests state-of-the-art models on DROP, a numerical MRC dataset, to see if they can handle passages that contain out-of-range numbers. One of the key findings is that the models fail to extrapolate to unseen numbers. Presenting numbers as digit-by-digit input to the model, we also propose the E-digit number form that alleviates the lack of extrapolation in models and reveals the need to treat numbers differently from regular words in the text. Our work provides a valuable insight into the numerical MRC models and the way to represent number forms in MRC.- Anthology ID:
- 2021.emnlp-main.563
- Volume:
- Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
- Month:
- November
- Year:
- 2021
- Address:
- Online and Punta Cana, Dominican Republic
- Editors:
- Marie-Francine Moens, Xuanjing Huang, Lucia Specia, Scott Wen-tau Yih
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 7031–7037
- Language:
- URL:
- https://aclanthology.org/2021.emnlp-main.563
- DOI:
- 10.18653/v1/2021.emnlp-main.563
- Cite (ACL):
- Jeonghwan Kim, Giwon Hong, Kyung-min Kim, Junmo Kang, and Sung-Hyon Myaeng. 2021. Have You Seen That Number? Investigating Extrapolation in Question Answering Models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7031–7037, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
- Cite (Informal):
- Have You Seen That Number? Investigating Extrapolation in Question Answering Models (Kim et al., EMNLP 2021)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2021.emnlp-main.563.pdf
- Data
- DROP