@inproceedings{kuksenok-martyniv-2019-evaluation,
    title = "Evaluation and Improvement of Chatbot Text Classification Data Quality Using Plausible Negative Examples",
    author = "Kuksenok, Kit  and
      Martyniv, Andriy",
    editor = "Chen, Yun-Nung  and
      Bedrax-Weiss, Tania  and
      Hakkani-Tur, Dilek  and
      Kumar, Anuj  and
      Lewis, Mike  and
      Luong, Thang-Minh  and
      Su, Pei-Hao  and
      Wen, Tsung-Hsien",
    booktitle = "Proceedings of the First Workshop on NLP for Conversational AI",
    month = aug,
    year = "2019",
    address = "Florence, Italy",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/W19-4110/",
    doi = "10.18653/v1/W19-4110",
    pages = "87--95",
    abstract = "We describe and validate a metric for estimating multi-class classifier performance based on cross-validation and adapted for improvement of small, unbalanced natural-language datasets used in chatbot design. Our experiences draw upon building recruitment chatbots that mediate communication between job-seekers and recruiters by exposing the ML/NLP dataset to the recruiting team. Evaluation approaches must be understandable to various stakeholders, and useful for improving chatbot performance. The metric, nex-cv, uses negative examples in the evaluation of text classification, and fulfils three requirements. First, it is actionable: it can be used by non-developer staff. Second, it is not overly optimistic compared to human ratings, making it a fast method for comparing classifiers. Third, it allows model-agnostic comparison, making it useful for comparing systems despite implementation differences. We validate the metric based on seven recruitment-domain datasets in English and German over the course of one year."
}