@inproceedings{yu-etal-2025-sequential,
    title = "Sequential-{NIAH}: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contexts",
    author = "Yu, Yifei  and
      Zhang, Qian-Wen  and
      Qiao, Lingfeng  and
      Yin, Di  and
      Li, Fang  and
      Wang, Jie  and
      Xi, Chen Zeng  and
      Zheng, Suncong  and
      Liang, Xiaolong  and
      Sun, Xing",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.emnlp-main.1497/",
    doi = "10.18653/v1/2025.emnlp-main.1497",
    pages = "29438--29456",
    ISBN = "979-8-89176-332-6",
    abstract = "Evaluating the ability of large language models (LLMs) to process lengthy contexts is critical, especially for retrieving query-relevant information embedded within them. We introduce Sequential-NIAH, a benchmark specifically designed to evaluate the capability of LLMs to extract sequential information items (known as \textit{needles}) from long contexts. The benchmark includes three needle generation pipelines: synthetic-temporal, real-temporal, and real-logical orders, with context lengths ranging from 8K to 128K, which comprises 14,000 samples (2,000 for testing). To facilitate the evaluation of this benchmark, we trained an evaluation model that assesses the correctness of LLM responses by comparing their completeness and sequential consistency against the ground truth, which provides a more reliable evaluation metric than GPT-4 or Claude. We conducted experiments on six well-known LLMs, revealing that even the best-performing model achieved a maximum accuracy of only 63.50{\%} on test set of this benchmark. Further analysis highlights the growing challenges posed by increasing the context length or the number of needles, underscoring substantial room for improvement of LLMs. Additionally, noise analysis validates the reliability and challenge of the benchmark, making Sequential-NIAH an important reference for advancing research on long text information extraction capabilities of LLMs."
}