@inproceedings{qin-etal-2025-relevant,
    title = "Relevant or Random: Can {LLM}s Truly Perform Analogical Reasoning?",
    author = "Qin, Chengwei  and
      Xia, Wenhan  and
      Wang, Tan  and
      Jiao, Fangkai  and
      Hu, Yuchen  and
      Ding, Bosheng  and
      Chen, Ruirui  and
      Joty, Shafiq",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/transition-to-people-yaml/2025.findings-acl.1230/",
    doi = "10.18653/v1/2025.findings-acl.1230",
    pages = "23993--24010",
    ISBN = "979-8-89176-256-5",
    abstract = "Analogical reasoning is a unique ability of humans to address unfamiliar challenges by transferring strategies from relevant past experiences. One key finding in psychology is that compared with irrelevant past experiences, recalling relevant ones can help humans better handle new tasks. Coincidentally, the NLP community has also recently found that self-generating relevant examples in the context can help large language models (LLMs) better solve a given problem than hand-crafted prompts. However, it is yet not clear whether relevance is the key factor eliciting such capability, i.e., can LLMs benefit more from self-generated relevant examples than irrelevant ones? In this work, we systematically explore whether LLMs can truly perform analogical reasoning on a diverse set of reasoning tasks. With extensive experiments and analysis, we show that self-generated random examples can surprisingly achieve comparable or even better performance on certain tasks, e.g., 4{\%} performance boost on GSM8K with random biological examples. We find that the accuracy of self-generated examples is the key factor and subsequently design two novel methods with improved performance and significantly reduced inference costs. Overall, we aim to advance a deeper understanding of LLM analogical reasoning and hope this work stimulates further research in the design of self-generated contexts."
}