Calibrating Imbalanced Classifiers with Focal Loss: An Empirical Study
Cheng Wang, Jorge Balazs, György Szarvas, Patrick Ernst, Lahari Poddar, Pavel Danchenko
Abstract
Imbalanced data distribution is a practical and common challenge in building production-level machine learning (ML) models in industry, where data usually exhibits long-tail distributions. For instance, in virtual AI Assistants, such as Google Assistant, Amazon Alexa and Apple Siri, the “play music” or “set timer” utterance is exposed to an order of magnitude more traffic than other skills. This can easily cause trained models to overfit to the majority classes, categories or intents, lead to model miscalibration. The uncalibrated models output unreliable (mostly overconfident) predictions, which are at high risk of affecting downstream decision-making systems. In this work, we study the calibration of production models in the industry use-case of predicting product return reason codes in customer service conversations of an online retail store; The returns reasons also exhibit class imbalance. To alleviate the resulting miscalibration in the production ML model, we streamline the model development and deployment using focal loss (CITATION).We empirically show the effectiveness of model training with focal loss in learning better calibrated models, as compared to standard cross-entropy loss. Better calibration, in turn, enables better control of the precision-recall trade-off for the models deployed in production.- Anthology ID:
- 2022.emnlp-industry.14
- Volume:
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, UAE
- Editors:
- Yunyao Li, Angeliki Lazaridou
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 145–153
- Language:
- URL:
- https://aclanthology.org/2022.emnlp-industry.14
- DOI:
- 10.18653/v1/2022.emnlp-industry.14
- Cite (ACL):
- Cheng Wang, Jorge Balazs, György Szarvas, Patrick Ernst, Lahari Poddar, and Pavel Danchenko. 2022. Calibrating Imbalanced Classifiers with Focal Loss: An Empirical Study. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 145–153, Abu Dhabi, UAE. Association for Computational Linguistics.
- Cite (Informal):
- Calibrating Imbalanced Classifiers with Focal Loss: An Empirical Study (Wang et al., EMNLP 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2022.emnlp-industry.14.pdf