Beyond Single-Shot: Multi-step Tool Retrieval via Query Planning

Wei Fang, James R. Glass


Abstract
LLM agents operating over massive, dynamic tool libraries rely on effective retrieval, yet standard single-shot dense retrievers struggle with complex requests. These failures primarily stem from the disconnect between abstract user goals and technical documentation, and the limited capacity of fixed-size embeddings to model combinatorial tool compositions. To address these challenges, we propose ToolQP, a lightweight framework that models retrieval as iterative query planning. Instead of single-shot matching, ToolQP decomposes instructions into sub-tasks and dynamically generates queries to interact with the retriever, effectively bridging the semantic gap by targeting the specific sub-tasks required for composition. We train ToolQP using synthetic query trajectories followed by optimization with Reinforcement Learning with Verifiable Rewards (RLVR). Experiments demonstrate that ToolQP achieves state-of-the-art performance, exhibiting superior zero-shot generalization, robustness across diverse retrievers, and significant improvements in downstream agentic execution.
Anthology ID:
2026.findings-acl.2090
Volume:
Findings of the Association for Computational Linguistics: ACL 2026
Month:
July
Year:
2026
Address:
San Diego, California, United States
Editors:
Maria Liakata, Viviane P. Moreira, Jiajun Zhang, David Jurgens
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
42119–42144
Language:
URL:
https://preview.aclanthology.org/ingest-acl/2026.findings-acl.2090/
DOI:
Bibkey:
Cite (ACL):
Wei Fang and James R. Glass. 2026. Beyond Single-Shot: Multi-step Tool Retrieval via Query Planning. In Findings of the Association for Computational Linguistics: ACL 2026, pages 42119–42144, San Diego, California, United States. Association for Computational Linguistics.
Cite (Informal):
Beyond Single-Shot: Multi-step Tool Retrieval via Query Planning (Fang & Glass, Findings 2026)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-acl/2026.findings-acl.2090.pdf
Checklist:
 2026.findings-acl.2090.checklist.pdf