@inproceedings{tian-etal-2025-cmt,
    title = "{CMT}-Eval: A Novel {C}hinese Multi-turn Dialogue Evaluation Dataset Addressing Real-world Conversational Challenges",
    author = "Tian, Siyu  and
      Mo, Kaijie  and
      Wang, Yupei  and
      Hu, Renfen",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.992/",
    doi = "10.18653/v1/2025.findings-emnlp.992",
    pages = "18279--18303",
    ISBN = "979-8-89176-335-7",
    abstract = "Multi-turn dialogue is a key paradigm for interaction between users and Large Language Models (LLMs). However, existing evaluation benchmarks fail to capture users' evolving needs and how their diverse conversation styles affect the dialogue flow. To address these limitations, we propose CMT-Eval, the first dedicated dataset for fine-grained evaluation of Chinese multi-turn dialogue systems. Built upon a linguistic theory-driven Speech Act Framework, diverse user personas, and varied conversational challenges, CMT-Eval comprises 596 high-quality dialogues with 4,431 turns, simulating realistic, multifaceted, and challenging conversations. Experiments reveal that models struggle with specific speech acts, user personas, and complex scenarios, highlighting the effectiveness of CMT-Eval in assessing LLMs' multi-turn dialogue capabilities and providing valuable insights for their enhancement. The dataset, code, and prompts are available at \url{https://github.com/hejaida/CMT-Eval}."
}