@inproceedings{vigel-etal-2025-self,
    title = "Self Knowledge-Tracing for Tool Use ({SKT}-Tool): Helping {LLM} Agents Understand Their Capabilities in Tool Use",
    author = "Vigel, Joshua  and
      Cai, Renpei  and
      Chen, Eleanor  and
      Neema, Anish  and
      Liao, Austen  and
      Zhu, Kevin  and
      O{'}brien, Sean",
    editor = "Drozd, Aleksandr  and
      Sedoc, Jo{\~a}o  and
      Tafreshi, Shabnam  and
      Akula, Arjun  and
      Shu, Raphael",
    booktitle = "The Sixth Workshop on Insights from Negative Results in NLP",
    month = may,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.insights-1.14/",
    pages = "150--156",
    ISBN = "979-8-89176-240-4",
    abstract = "Large Language Models (LLMs) enhanced with tool use and APIs improve task performance but often misuse them, leading to inefficiency and unnecessary cost. We propose Self Knowledge-Tracing for Tool Use (SKT-Tool), a method enabling LLMs to assess their capabilities and make informed API usage decisions using knowledge tracing (KT). Our teacher-student framework helps LLMs optimize API calls in real-time without fine-tuning. Experiments across multiple datasets show that SKT-Tool significantly reduces API calls while maintaining accuracy, offering a scalable and cost-effective solution for tool-augmented LLMs. We conclude by analyzing shortcomings in this method and identifying directions for future work."
}