@inproceedings{zhang-etal-2025-designed,
    title = "Have We Designed Generalizable Structural Knowledge Promptings? Systematic Evaluation and Rethinking",
    author = "Zhang, Yichi  and
      Chen, Zhuo  and
      Guo, Lingbing  and
      Xu, Yajing  and
      Chen, Shaokai  and
      Sun, Mengshu  and
      Hu, Binbin  and
      Zhang, Zhiqiang  and
      Liang, Lei  and
      Zhang, Wen  and
      Chen, Huajun",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/acl25-workshop-ingestion/2025.acl-long.110/",
    pages = "2210--2226",
    ISBN = "979-8-89176-251-0",
    abstract = "Large language models (LLMs) have demonstrated exceptional performance in text generation within current NLP research. However, the lack of factual accuracy is still a dark cloud hanging over the LLM skyscraper. Structural knowledge prompting (SKP) is a prominent paradigm to integrate external knowledge into LLMs by incorporating structural representations, achieving state-of-the-art results in many knowledge-intensive tasks. However, existing methods often focus on specific problems, lacking a comprehensive exploration of the generalization and capability boundaries of SKP. This paper aims to evaluate and rethink the generalization capability of the SKP paradigm from four perspectives including Granularity, Transferability, Scalability, and Universality. To provide a thorough evaluation, we introduce a novel multi-granular, multi-level benchmark called SUBARU, consisting of 9 different tasks with varying levels of granularity and difficulty. Through extensive experiments, we draw key conclusions regarding the generalization of SKP, offering insights to guide the future development and extension of the SKP paradigm."
}