@inproceedings{duan-etal-2026-chunks,
    title = "Chunks as Arms: Multi-Armed Bandit-Guided Sampling for Long-Context {LLM} Preference Optimization",
    author = "Duan, Shaohua  and
      Huang, Pengcheng  and
      Li, Xinze  and
      Liu, Zhenghao  and
      Yi, Xiaoyuan  and
      Yan, Yukun  and
      Wang, Shuo  and
      Gu, Yu  and
      Yu, Ge  and
      Sun, Maosong",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Proceedings of the 64th Annual Meeting of the {A}ssociation for {C}omputational {L}inguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-acl/2026.acl-long.161/",
    pages = "3559--3575",
    ISBN = "979-8-89176-390-6",
    abstract = "Long-context modeling is critical for a wide range of real-world tasks, including long-context question answering, summarization, and complex reasoning tasks. Recent studies have explored fine-tuning Large Language Models (LLMs) with synthetic data to enhance their long-context capabilities. However, the effectiveness of such approaches is often limited by the low diversity and factual inconsistencies in the generated data. To address these challenges, we propose LongMab, a novel framework that leverages a Multi-Armed Bandit (MAB) rollout strategy to identify the most informative chunks from the given long context for sampling high-quality and diverse responses and constructing preference data pairs for Direct Preference Optimization (DPO) training. Specifically, we treat context chunks as arms of MAB, select chunks based on their expected reward scores to input into LLMs to generate responses, and iteratively update these scores based on reward feedback. Both exploration and exploitation during the rollout process enable the LLM to focus on the most relevant context segments, thereby generating and collecting high-quality and diverse responses. Experimental results on both Llama and Qwen show the effectiveness of LongMab by achieving more than a 4{\%} improvement on long-context reasoning benchmarks. All data and code will be released on https://github.com/NEUIR/LongMab-PO."
}