@inproceedings{jung-etal-2025-diatool,
    title = "{D}ia{T}ool-{DPO}: Multi-Turn Direct Preference Optimization for Tool-Augmented Large Language Models",
    author = "Jung, Sunghee  and
      Lee, Donghun  and
      Lee, Shinbok  and
      Seo, Gaeun  and
      Lee, Daniel  and
      Ko, Byeongil  and
      Cho, Junrae  and
      Kim, Kihyun  and
      Kim, EungGyun  and
      Shin, Myeongcheol",
    editor = "B{\'e}chet, Fr{\'e}d{\'e}ric  and
      Lef{\`e}vre, Fabrice  and
      Asher, Nicholas  and
      Kim, Seokhwan  and
      Merlin, Teva",
    booktitle = "Proceedings of the 26th Annual Meeting of the Special Interest Group on Discourse and Dialogue",
    month = aug,
    year = "2025",
    address = "Avignon, France",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/corrections-2025-10/2025.sigdial-1.32/",
    pages = "397--416",
    abstract = "Tool-Augmented Larage Language Models (TA-LLMs) have shown promise in real-world applications, but face challenges in handling incomplete queries and out-of-scope requests. While existing approaches rely mainly on Supervised Fine-Tuning with expert trajectories, we propose DiaTool-DPO, a novel method that enhances TA-LLM{'}s dialogue capabilities through Direct Preference Optimization. We model TA-LLM interactions as a Markov Decision Process with 5 distinct dialogue states and categorize user queries into 3 types based on their state transition trajectories. We automatically construct paired trajectory datasets of correct and incorrect dialogue flows and introduce a specialized objective loss for dialogue control. Our comprehensive evaluation demonstrates that DiaTool-DPO approaches GPT-4o{'}s performance (94.8{\%} in information gathering, 91{\%} in tool call rejection) with substantial improvements over baseline (44{\%} and 9.6{\%} respectively) while maintaining core functionality. Our approach opens new possibilities for developing TA-LLMs that can handle diverse real-world scenarios without requiring additional expert demonstrations or human labeling."
}