@inproceedings{huang-etal-2025-ttpa,
    title = "{TTPA}: Token-level Tool-use Preference Alignment Training Framework with Fine-grained Evaluation",
    author = "Huang, Chengrui  and
      Gao, Shen  and
      Shi, Zhengliang  and
      Wang, Dongsheng  and
      Shang, Shuo",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.882/",
    doi = "10.18653/v1/2025.findings-emnlp.882",
    pages = "16240--16255",
    ISBN = "979-8-89176-335-7",
    abstract = "Existing tool-learning methods usually rely on supervised fine-tuning, they often overlook fine-grained optimization of internal tool call details, leading to limitations in preference alignment and error discrimination. To overcome these challenges, we propose **T**oken-level **T**ool-use **P**reference **A**lignment Training Framework (TTPA), a training paradigm for constructing token-level tool-use preference datasets that align LLMs with fine-grained preferences using a novel error-oriented scoring mechanism. TTPA first introduces reversed dataset construction, a method for creating high-quality, multi-turn tool-use datasets by reversing the generation flow. Additionally, we propose {\_}Preference Oriented Tool-use Dataset Construction{\_} to capture fine-grained preferences by modeling token-level differences during generation. To address biases in scoring, we introduce the {\_}Error-oriented Scoring Mechanism{\_}, which quantifies tool-call errors and can be used as a training signal. Extensive experiments on three diverse benchmark datasets demonstrate that TTPA significantly improves tool-using performance while showing strong generalization ability across models and datasets."
}