Abstract
We describe and validate a metric for estimating multi-class classifier performance based on cross-validation and adapted for improvement of small, unbalanced natural-language datasets used in chatbot design. Our experiences draw upon building recruitment chatbots that mediate communication between job-seekers and recruiters by exposing the ML/NLP dataset to the recruiting team. Evaluation approaches must be understandable to various stakeholders, and useful for improving chatbot performance. The metric, nex-cv, uses negative examples in the evaluation of text classification, and fulfils three requirements. First, it is actionable: it can be used by non-developer staff. Second, it is not overly optimistic compared to human ratings, making it a fast method for comparing classifiers. Third, it allows model-agnostic comparison, making it useful for comparing systems despite implementation differences. We validate the metric based on seven recruitment-domain datasets in English and German over the course of one year.- Anthology ID:
- W19-4110
- Volume:
- Proceedings of the First Workshop on NLP for Conversational AI
- Month:
- August
- Year:
- 2019
- Address:
- Florence, Italy
- Editors:
- Yun-Nung Chen, Tania Bedrax-Weiss, Dilek Hakkani-Tur, Anuj Kumar, Mike Lewis, Thang-Minh Luong, Pei-Hao Su, Tsung-Hsien Wen
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 87–95
- Language:
- URL:
- https://aclanthology.org/W19-4110
- DOI:
- 10.18653/v1/W19-4110
- Cite (ACL):
- Kit Kuksenok and Andriy Martyniv. 2019. Evaluation and Improvement of Chatbot Text Classification Data Quality Using Plausible Negative Examples. In Proceedings of the First Workshop on NLP for Conversational AI, pages 87–95, Florence, Italy. Association for Computational Linguistics.
- Cite (Informal):
- Evaluation and Improvement of Chatbot Text Classification Data Quality Using Plausible Negative Examples (Kuksenok & Martyniv, ACL 2019)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/W19-4110.pdf
- Code
- jobpal/nex-cv