Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses
Erfan Sadeqi Azer, Daniel Khashabi, Ashish Sabharwal, Dan Roth
Abstract
Empirical research in Natural Language Processing (NLP) has adopted a narrow set of principles for assessing hypotheses, relying mainly on p-value computation, which suffers from several known issues. While alternative proposals have been well-debated and adopted in other fields, they remain rarely discussed or used within the NLP community. We address this gap by contrasting various hypothesis assessment techniques, especially those not commonly used in the field (such as evaluations based on Bayesian inference). Since these statistical techniques differ in the hypotheses they can support, we argue that practitioners should first decide their target hypothesis before choosing an assessment method. This is crucial because common fallacies, misconceptions, and misinterpretation surrounding hypothesis assessment methods often stem from a discrepancy between what one would like to claim versus what the method used actually assesses. Our survey reveals that these issues are omnipresent in the NLP research community. As a step forward, we provide best practices and guidelines tailored to NLP research, as well as an easy-to-use package for Bayesian assessment of hypotheses, complementing existing tools.- Anthology ID:
- 2020.acl-main.506
- Volume:
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Editors:
- Dan Jurafsky, Joyce Chai, Natalie Schluter, Joel Tetreault
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 5715–5725
- Language:
- URL:
- https://aclanthology.org/2020.acl-main.506
- DOI:
- 10.18653/v1/2020.acl-main.506
- Cite (ACL):
- Erfan Sadeqi Azer, Daniel Khashabi, Ashish Sabharwal, and Dan Roth. 2020. Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5715–5725, Online. Association for Computational Linguistics.
- Cite (Informal):
- Not All Claims are Created Equal: Choosing the Right Statistical Approach to Assess Hypotheses (Sadeqi Azer et al., ACL 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2020.acl-main.506.pdf
- Code
- allenai/HyBayes