@inproceedings{sato-etal-2025-key,
    title = "Key Challenges in Multimodal Task-Oriented Dialogue Systems: Insights from a Large Competition-Based Dataset",
    author = "Sato, Shiki  and
      Iwata, Shinji  and
      Hentona, Asahi  and
      Sasaki, Yuta  and
      Yamazaki, Takato  and
      Moriya, Shoji  and
      Ohagi, Masaya  and
      Kikuchi, Hirofumi  and
      Yang, Jie  and
      Qi, Zhiyang  and
      Kodama, Takashi  and
      Lee, Akinobu  and
      Komuro, Masato  and
      Nishikawa, Hiroyuki  and
      Makino, Ryosaku  and
      Minato, Takashi  and
      Sakai, Kurima  and
      Funayama, Tomo  and
      Funakoshi, Kotaro  and
      Usami, Mayumi  and
      Inaba, Michimasa  and
      Takahashi, Tetsuro  and
      Higashinaka, Ryuichiro",
    editor = "B{\'e}chet, Fr{\'e}d{\'e}ric  and
      Lef{\`e}vre, Fabrice  and
      Asher, Nicholas  and
      Kim, Seokhwan  and
      Merlin, Teva",
    booktitle = "Proceedings of the 26th Annual Meeting of the Special Interest Group on Discourse and Dialogue",
    month = aug,
    year = "2025",
    address = "Avignon, France",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/corrections-2025-10/2025.sigdial-1.36/",
    pages = "449--460",
    abstract = "Challenges in multimodal task-oriented dialogue between humans and systems, particularly those involving audio and visual interactions, have not been sufficiently explored or shared, forcing researchers to define improvement directions individually without a clearly shared roadmap. To address these challenges, we organized a competition for multimodal task-oriented dialogue systems and constructed a large competition-based dataset of 1,865 minutes of Japanese task-oriented dialogues. This dataset includes audio and visual interactions between diverse systems and human participants. After analyzing system behaviors identified as problematic by the human participants in questionnaire surveys and notable methods employed by the participating teams, we identified key challenges in multimodal task-oriented dialogue systems and discussed potential directions for overcoming these challenges."
}