Abstract
We provide a survey and empirical comparison of the state-of-the-art in neural selective classification for NLP tasks. We also provide a methodological blueprint, including a novel metric called refinement that provides a calibrated evaluation of confidence functions for selective prediction. Finally, we supply documented, open-source code to support the future development of selective prediction techniques.- Anthology ID:
- 2023.acl-long.437
- Volume:
- Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 7888–7899
- Language:
- URL:
- https://aclanthology.org/2023.acl-long.437
- DOI:
- 10.18653/v1/2023.acl-long.437
- Cite (ACL):
- Zhengyao Gu and Mark Hopkins. 2023. On the Evaluation of Neural Selective Prediction Methods for Natural Language Processing. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7888–7899, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- On the Evaluation of Neural Selective Prediction Methods for Natural Language Processing (Gu & Hopkins, ACL 2023)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2023.acl-long.437.pdf