Abstract
Modern NLP classifiers are known to return uncalibrated estimations of class posteriors. Existing methods for posterior calibration rescale the predicted probabilities but often have an adverse impact on final classification accuracy, thus leading to poorer generalization. We propose an end-to-end trained calibrator, Platt-Binning, that directly optimizes the objective while minimizing the difference between the predicted and empirical posterior probabilities. Our method leverages the sample efficiency of Platt scaling and the verification guarantees of histogram binning, thus not only reducing the calibration error but also improving task performance. In contrast to existing calibrators, we perform this efficient calibration during training. Empirical evaluation of benchmark NLP classification tasks echoes the efficacy of our proposal.- Anthology ID:
- 2022.findings-acl.290
- Volume:
- Findings of the Association for Computational Linguistics: ACL 2022
- Month:
- May
- Year:
- 2022
- Address:
- Dublin, Ireland
- Editors:
- Smaranda Muresan, Preslav Nakov, Aline Villavicencio
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 3673–3684
- Language:
- URL:
- https://preview.aclanthology.org/icon-24-ingestion/2022.findings-acl.290/
- DOI:
- 10.18653/v1/2022.findings-acl.290
- Cite (ACL):
- Rishabh Singh and Shirin Goshtasbpour. 2022. Platt-Bin: Efficient Posterior Calibrated Training for NLP Classifiers. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3673–3684, Dublin, Ireland. Association for Computational Linguistics.
- Cite (Informal):
- Platt-Bin: Efficient Posterior Calibrated Training for NLP Classifiers (Singh & Goshtasbpour, Findings 2022)
- PDF:
- https://preview.aclanthology.org/icon-24-ingestion/2022.findings-acl.290.pdf