@inproceedings{finch-etal-2021-went,
    title = "What Went Wrong? Explaining Overall Dialogue Quality through Utterance-Level Impacts",
    author = "Finch, James D.  and
      Finch, Sarah E.  and
      Choi, Jinho D.",
    editor = "Papangelis, Alexandros  and
      Budzianowski, Pawe{\l}  and
      Liu, Bing  and
      Nouri, Elnaz  and
      Rastogi, Abhinav  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 3rd Workshop on Natural Language Processing for Conversational AI",
    month = nov,
    year = "2021",
    address = "Online",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2021.nlp4convai-1.9/",
    doi = "10.18653/v1/2021.nlp4convai-1.9",
    pages = "93--101",
    abstract = "Improving user experience of a dialogue system often requires intensive developer effort to read conversation logs, run statistical analyses, and intuit the relative importance of system shortcomings. This paper presents a novel approach to automated analysis of conversation logs that learns the relationship between user-system interactions and overall dialogue quality. Unlike prior work on utterance-level quality prediction, our approach learns the impact of each interaction from the overall user rating without utterance-level annotation, allowing resultant model conclusions to be derived on the basis of empirical evidence and at low cost. Our model identifies interactions that have a strong correlation with the overall dialogue quality in a chatbot setting. Experiments show that the automated analysis from our model agrees with expert judgments, making this work the first to show that such weakly-supervised learning of utterance-level quality prediction is highly achievable."
}