@inproceedings{liming-etal-2024-fourth,
    title = "The Fourth Evaluation on {C}hinese Spatial Cognition",
    author = "Xiao, Liming  and
      Hu, Nan  and
      Zhan, Weidong  and
      Qin, Yuhang  and
      Deng, Sirui  and
      Sun, Chunhui  and
      Cai, Qixu  and
      Li, Nan",
    editor = "Hongfei, Lin  and
      Hongye, Tan  and
      Bin, Li",
    booktitle = "Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 3: Evaluations)",
    month = jul,
    year = "2024",
    address = "Taiyuan, China",
    publisher = "Chinese Information Processing Society of China",
    url = "https://preview.aclanthology.org/author-degibert/2024.ccl-3.14/",
    pages = "122--134",
    language = "eng",
    abstract = "``The Fourth Chinese Spatial Cognition Evaluation Task (SpaCE 2024) presents the first comprehensive Chinese benchmark to assess spatial semantic understanding and reasoning capabilities of Large Language Models (LLMs). It comprises five subtasks in the form of multiple-choice questions: (1) identifying spatial semantic roles; (2) retrieving spatial referents; (3) detecting spatial semantic anomalies; (4) recognizing synonymous spatial expression with different forms; (5) conducting spatial position reasoning. In addition to proposing new tasks, SpaCE 2024 applied a rule-based method to generate high-quality synthetic data with difficulty levels for the reasoning task. 12 teams submitted their models and results, and the top-performing team attained an accuracy of 60.24{\%}, suggesting that there is still significant room for current LLMs to improve, especially in tasks requiring high spatial cognitive processing.''"
}