@inproceedings{lee-etal-2025-loft,
    title = "{LOFT}: Scalable and More Realistic Long-Context Evaluation",
    author = "Lee, Jinhyuk  and
      Chen, Anthony  and
      Dai, Zhuyun  and
      Dua, Dheeru  and
      Sachan, Devendra Singh  and
      Boratko, Michael  and
      Luan, Yi  and
      Arnold, S{\'e}b  and
      Perot, Vincent  and
      Dalmia, Siddharth  and
      Hu, Hexiang  and
      Lin, Xudong  and
      Pasupat, Panupong  and
      Amini, Aida  and
      Cole, Jeremy R.  and
      Riedel, Sebastian  and
      Naim, Iftekhar  and
      Chang, Ming-Wei  and
      Guu, Kelvin",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.findings-naacl.374/",
    pages = "6698--6723",
    ISBN = "979-8-89176-195-7",
    abstract = "Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases. Leveraging LCLMs' ability to natively ingest and process entire corpora of information offers numerous advantages. It enhances user-friendliness by eliminating the need for specialized knowledge of tools, provides robust end-to-end modeling that minimizes cascading errors in complex pipelines, and allows for the application of sophisticated prompting techniques across the entire system. To assess this paradigm shift, we introduce LOFT, a benchmark of real-world tasks requiring context up to millions of tokens designed to evaluate LCLMs' performance on in-context retrieval and reasoning. Our findings reveal LCLMs' surprising ability to rival state-of-the-art retrieval and RAG systems, despite never having been explicitly trained for these tasks. However, LCLMs still face challenges in areas like compositional reasoning that are required in SQL-like tasks. Notably, prompting strategies significantly influence performance, emphasizing the need for continued research. Overall, LOFT provides a rigorous testing ground for LCLMs, showcasing their capabilities to tackle existing paradigms."
}