Abstract
Open-source software packages for language processing often include stop word lists. Users may apply them without awareness of their surprising omissions (e.g. “hasn’t” but not “hadn’t”) and inclusions (“computer”), or their incompatibility with a particular tokenizer. Motivated by issues raised about the Scikit-learn stop list, we investigate variation among and consistency within 52 popular English-language stop lists, and propose strategies for mitigating these issues.- Anthology ID:
- W18-2502
- Volume:
- Proceedings of Workshop for NLP Open Source Software (NLP-OSS)
- Month:
- July
- Year:
- 2018
- Address:
- Melbourne, Australia
- Venue:
- NLPOSS
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 7–12
- Language:
- URL:
- https://aclanthology.org/W18-2502
- DOI:
- 10.18653/v1/W18-2502
- Cite (ACL):
- Joel Nothman, Hanmin Qin, and Roman Yurchak. 2018. Stop Word Lists in Free Open-source Software Packages. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 7–12, Melbourne, Australia. Association for Computational Linguistics.
- Cite (Informal):
- Stop Word Lists in Free Open-source Software Packages (Nothman et al., NLPOSS 2018)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/W18-2502.pdf