Abstract
Recent literature has seen growing interest in using black-box strategies like for testing the behavior of NLP models. Research on white-box testing has developed a number of methods for evaluatinghow thoroughly the internal behavior of deep models is tested, but they are not applicableto NLP models.We propose a set of white-box testing methods that are customized for transformer-based NLP models.These include MASK NEURON COVERAGE (MNCOVER) that measures how thoroughlythe attention layers in models are exercised during testing.We show that MNCOVER can refine testing suites generated by CheckList by substantiallyreduce them in size, for more than 60% on average, while retaining failing tests – thereby concentrating the faultdetection power of the test suite.Further we show how can be used to guide CheckList input generation,evaluate alternative NLP testing methods, and drive data augmentation to improve accuracy.- Anthology ID:
- 2022.findings-naacl.116
- Volume:
- Findings of the Association for Computational Linguistics: NAACL 2022
- Month:
- July
- Year:
- 2022
- Address:
- Seattle, United States
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1547–1558
- Language:
- URL:
- https://aclanthology.org/2022.findings-naacl.116
- DOI:
- 10.18653/v1/2022.findings-naacl.116
- Cite (ACL):
- Arshdeep Sekhon, Yangfeng Ji, Matthew Dwyer, and Yanjun Qi. 2022. White-box Testing of NLP models with Mask Neuron Coverage. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 1547–1558, Seattle, United States. Association for Computational Linguistics.
- Cite (Informal):
- White-box Testing of NLP models with Mask Neuron Coverage (Sekhon et al., Findings 2022)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2022.findings-naacl.116.pdf