@inproceedings{luo-etal-2025-turnback,
    title = "{T}urn{B}ack: A Geospatial Route Cognition Benchmark for Large Language Models through Reverse Route",
    author = "Luo, Hongyi  and
      Cheng, Qing  and
      Matos, Daniel  and
      Gadi, Hari Krishna  and
      Zhang, Yanfeng  and
      Liu, Lu  and
      Wang, Yongliang  and
      Zeller, Niclas  and
      Cremers, Daniel  and
      Meng, Liqiu",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.440/",
    pages = "8721--8740",
    ISBN = "979-8-89176-332-6",
    abstract = "Humans can interpret geospatial information through natural language, while the geospatial cognition capabilities of Large Language Models (LLMs) remain underexplored. Prior research in this domain has been constrained by non-quantifiable metrics, limited evaluation datasets; unclear research hierarchies further compound these limitations. Therefore, we propose a scalable benchmark and conduct a comprehensive evaluation of the geospatial route cognition of LLMs. We create a large-scale evaluation dataset comprised of 36000 routes from 12 metropolises. Then, we introduce PathBuilder, a novel tool for converting natural language instructions into navigation routes, and vice versa, bridging the gap between geospatial information and natural language. Finally, we propose a new evaluation framework and metrics to rigorously assess 9 state-of-the-art (SOTA) LLMs, on the task of route reversal. The benchmark reveals that LLMs exhibit limited ability to reverse routes: most of the reverse routes neither return to the starting point nor are similar to the optimal route. Additionally, LLMs face challenges such as low robustness in route generation and high confidence for their incorrect answers."
}