@inproceedings{jung-etal-2025-llms,
    title = "Can {LLM}s Truly Plan? A Comprehensive Evaluation of Planning Capabilities",
    author = "Jung, Gayeon  and
      Lim, HyeonSeok  and
      Kim, Minjun  and
      Lim, Joon-ho  and
      Lim, KyungTae  and
      Kim, Hansaem",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.702/",
    doi = "10.18653/v1/2025.findings-emnlp.702",
    pages = "13069--13084",
    ISBN = "979-8-89176-335-7",
    abstract = "The existing assessments of planning capabilities of large language models (LLMs) remain largely limited to single-language or specific representation formats. To address this gap, we introduce the Multi-Plan benchmark comprising 204 multilingual and multi-format travel planning scenarios. In experimental results obtained with state-of-the-art LLMs, the Multi-Plan benchmark effectively highlights the performance disparities among models, notably showing superior results for reasoning-specialized models. Interestingly, language differences exhibited minimal impact, whereas mathematically structured representations significantly improved planning accuracy for most models, underscoring the crucial role of the input format. These findings enhance our understanding of planning abilities of LLMs, offer valuable insights for future research, and emphasize the need for more sophisticated AI evaluation methods. This dataset is publicly available at http://huggingface.co/datasets/Bllossom/Multi-Plan."
}