Abstract
This paper adds to the ongoing discussion in the natural language processing community on how to choose a good development set. Motivated by the real-life necessity of applying machine learning models to different data distributions, we propose a clustering-based data splitting algorithm. It creates development (or test) sets which are lexically different from the training data while ensuring similar label distributions. Hence, we are able to create challenging cross-validation evaluation setups while abstracting away from performance differences resulting from label distribution shifts between training and test data. In addition, we present a Python-based tool for analyzing and visualizing data split characteristics and model performance. We illustrate the workings and results of our approach using a sentiment analysis and a patent classification task.- Anthology ID:
- 2020.eval4nlp-1.15
- Volume:
- Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Venue:
- Eval4NLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 155–163
- Language:
- URL:
- https://aclanthology.org/2020.eval4nlp-1.15
- DOI:
- 10.18653/v1/2020.eval4nlp-1.15
- Cite (ACL):
- Hanna Wecker, Annemarie Friedrich, and Heike Adel. 2020. ClusterDataSplit: Exploring Challenging Clustering-Based Data Splits for Model Performance Evaluation. In Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems, pages 155–163, Online. Association for Computational Linguistics.
- Cite (Informal):
- ClusterDataSplit: Exploring Challenging Clustering-Based Data Splits for Model Performance Evaluation (Wecker et al., Eval4NLP 2020)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2020.eval4nlp-1.15.pdf
- Code
- boschresearch/clusterdatasplit_eval4nlp-2020
- Data
- SST