@inproceedings{miyazawa-sato-2026-evaluation,
    title = "Evaluation of Paralinguistic-Aware Spoken Dialogue Systems using Next-Utterance Classification",
    author = "Miyazawa, Kouki  and
      Sato, Yoshinao",
    editor = "Choi, Jinho D.  and
      Chen, Yun-Nung  and
      Funakoshi, Kotaro  and
      Emami, Ali",
    booktitle = "Proceedings of the 27th Annual Meeting of the Special Interest Group on Discourse and Dialogue",
    month = aug,
    year = "2026",
    address = "Atlanta, Georgia, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/revision-workflow/2026.sigdial-1.50/",
    pages = "711--719",
    abstract = "In spoken dialogues, paralinguistic cues frequently convey crucial information not captured by linguistic content alone. However, conventional spoken dialogue systems (SDSs), which typically comprise a cascade of an automatic speech recognition model and a large language model (LLM), lack the ability to recognize paralinguistic cues. Relying solely on transcribed text, paralinguistic-agnostic SDSs often cause dialogue breakdowns. To address this difficulty, integrating a paralinguistic recognition model into SDSs is essential. Therefore, this study focuses on evaluating such paralinguistic-aware SDSs. To this end, we propose a corpus-based evaluation method utilizing next-utterance classification as an automated alternative to human evaluation. Specifically, an LLM is tasked with predicting the dialogue act of the subsequent utterance given a transcribed dialogue history, comparing scenarios with and without paralinguistic attitude classes. Our experiments demonstrate that incorporating a paralinguistic attitude recognition model improves prediction performance, as measured by the mean reciprocal rank. Furthermore, we assessed the alignment of our proposed corpus-based evaluation method with subjective human evaluations. The results confirmed the validity of the proposed method as a reliable proxy for human evaluation, demonstrating its correlation with human rankings and agreement with human preferences in pairwise comparisons. Taken together, this work highlights that integrating paralinguistic recognition is a crucial step toward realizing more robust and natural spoken dialogue systems."
}