ARES: A Reading Comprehension Ensembling Service
Anthony Ferritto, Lin Pan, Rishav Chakravarti, Salim Roukos, Radu Florian, J. William Murdock, Avi Sil
Abstract
We introduce ARES (A Reading Comprehension Ensembling Service): a novel Machine Reading Comprehension (MRC) demonstration system which utilizes an ensemble of models to increase F1 by 2.3 points. While many of the top leaderboard submissions in popular MRC benchmarks such as the Stanford Question Answering Dataset (SQuAD) and Natural Questions (NQ) use model ensembles, the accompanying papers do not publish their ensembling strategies. In this work, we detail and evaluate various ensembling strategies using the NQ dataset. ARES leverages the CFO (Chakravarti et al., 2019) and ReactJS distributed frameworks to provide a scalable interactive Question Answering experience that capitalizes on the agreement (or lack thereof) between models to improve the answer visualization experience.- Anthology ID:
- 2020.emnlp-demos.5
- Volume:
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations
- Month:
- October
- Year:
- 2020
- Address:
- Online
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 31–37
- Language:
- URL:
- https://aclanthology.org/2020.emnlp-demos.5
- DOI:
- 10.18653/v1/2020.emnlp-demos.5
- Cite (ACL):
- Anthony Ferritto, Lin Pan, Rishav Chakravarti, Salim Roukos, Radu Florian, J. William Murdock, and Avi Sil. 2020. ARES: A Reading Comprehension Ensembling Service. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 31–37, Online. Association for Computational Linguistics.
- Cite (Informal):
- ARES: A Reading Comprehension Ensembling Service (Ferritto et al., EMNLP 2020)
- PDF:
- https://preview.aclanthology.org/remove-xml-comments/2020.emnlp-demos.5.pdf
- Data
- Natural Questions, SQuAD