Abstract
Most tools for natural language processing today are based on machine learning and come with pre-trained models. In addition, third-parties provide pre-trained models for popular NLP tools. The predictive power and accuracy of these tools depends on the quality of these models. Downstream researchers often base their results on pre-trained models instead of training their own. Consequently, pre-trained models are an essential resource to our community. However, to be best of our knowledge, no systematic study of pre-trained models has been conducted so far. This paper reports on the analysis of 274 pre-models for six NLP tools and four potential causes of problems: encoding, tokenization, normalization and change over time. The analysis is implemented in the open source tool Model Investigator. Our work 1) allows model consumers to better assess whether a model is suitable for their task, 2) enables tool and model creators to sanity-check their models before distributing them, and 3) enables improvements in tool interoperability by performing automatic adjustments of normalization or other pre-processing based on the models used.- Anthology ID:
- W16-5203
- Volume:
- Proceedings of the Third International Workshop on Worldwide Language Service Infrastructure and Second Workshop on Open Infrastructures and Analysis Frameworks for Human Language Technologies (WLSI/OIAF4HLT2016)
- Month:
- December
- Year:
- 2016
- Address:
- Osaka, Japan
- Editors:
- Yohei Murakami, Donghui Lin, Nancy Ide, James Pustejovsky
- Venue:
- OIAF4HLT
- SIG:
- Publisher:
- The COLING 2016 Organizing Committee
- Note:
- Pages:
- 19–27
- Language:
- URL:
- https://aclanthology.org/W16-5203
- DOI:
- Cite (ACL):
- Richard Eckart de Castilho. 2016. Automatic Analysis of Flaws in Pre-Trained NLP Models. In Proceedings of the Third International Workshop on Worldwide Language Service Infrastructure and Second Workshop on Open Infrastructures and Analysis Frameworks for Human Language Technologies (WLSI/OIAF4HLT2016), pages 19–27, Osaka, Japan. The COLING 2016 Organizing Committee.
- Cite (Informal):
- Automatic Analysis of Flaws in Pre-Trained NLP Models (Eckart de Castilho, OIAF4HLT 2016)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/W16-5203.pdf
- Code
- UKPLab/coling2016-modelinspector