Nicholas Comment


2022

pdf
Iterative Stratified Testing and Measurement for Automated Model Updates
Elizabeth Dekeyser | Nicholas Comment | Shermin Pei | Rajat Kumar | Shruti Rai | Fengtao Wu | Lisa Haverty | Kanna Shimizu
Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: Industry Track

Automating updates to machine learning systems is an important but understudied challenge in AutoML. The high model variance of many cutting-edge deep learning architectures means that retraining a model provides no guarantee of accurate inference on all sample types. To address this concern, we present Automated Data-Shape Stratified Model Updates (ADSMU), a novel framework that relies on iterative model building coupled with data-shape stratified model testing and improvement. Using ADSMU, we observed a 26% (relative) improvement in accuracy for new model use cases on a large-scale NLU system, compared to a naive (manually) retrained baseline and current cutting-edge methods.