@inproceedings{qiu-etal-2025-eliciting,
    title = "Eliciting In-context Retrieval and Reasoning for Long-context Large Language Models",
    author = "Qiu, Yifu  and
      Embar, Varun R.  and
      Zhang, Yizhe  and
      Jaitly, Navdeep  and
      Cohen, Shay B  and
      Han, Benjamin",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/mtsummit-25-ingestion/2025.findings-acl.165/",
    doi = "10.18653/v1/2025.findings-acl.165",
    pages = "3176--3192",
    ISBN = "979-8-89176-256-5",
    abstract = "Recent advancements in long-context language models (LCLMs) promise to transform Retrieval-Augmented Generation (RAG) by simplifying pipelines. With their expanded context windows, LCLMs can process entire knowledge bases and perform retrieval and reasoning directly {--} a capability we define as In-Context Retrieval and Reasoning (ICR{\textasciicircum}2). However, existing benchmarks like LOFT often overestimate LCLM performance by providing overly simplified contexts. To address this, we introduce ICR{\textasciicircum}2, a benchmark that evaluates LCLMs in more realistic scenarios by including confounding passages retrieved with strong retrievers. We then propose three methods to enhance LCLM performance: (1) retrieve-then-generate fine-tuning, (2) retrieval-attention-probing, which uses attention heads to filter and de-noise long contexts during decoding, and (3) joint retrieval head training alongside the generation head. Our evaluation of five well-known LCLMs on LOFT and ICR{\textasciicircum}2 demonstrates significant gains with our best approach applied to Mistral-7B: +17 and +15 points by Exact Match on LOFT, and +13 and +2 points on ICR{\textasciicircum}2, compared to vanilla RAG and supervised fine-tuning, respectively. It even outperforms GPT-4-Turbo on most tasks despite being a much smaller model."
}