@inproceedings{zhang-etal-2025-re3syn,
    title = "Re$^{3}$Syn: A Dependency-Based Data Synthesis Framework for Long-Context Post-training",
    author = "Zhang, Zhiyang  and
      Liu, Ziqiang  and
      Wang, Huiming  and
      Shan, Renke  and
      Kuang, Li  and
      Wang, Lu  and
      Soh, De Wen",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1518/",
    pages = "31468--31480",
    ISBN = "979-8-89176-251-0",
    abstract = "An important trend in the realm of large language models (LLMs) is the development of longer context windows. However, training LLMs with long context windows to acquire the capability of effectively modeling lengthy inputs is often hindered by the scarcity of naturally long-context data. Existing methods for constructing long-context data by concatenating short documents have overlooked a crucial characteristic of long-context data quality, namely semantic dependency. In this paper, we propose a novel framework called Retrieval, Dependency Recognition, and Reorder for data synthesis (Re$^{3}$Syn), which leverages semantic similarity to retrieve relevant documents and form several batches. Within each batch, the framework comprehensively recognizes dependency and utilizes them, along with a reorder algorithm, to organize the short documents into coherent long-context data. Comprehensive experiment on multiple benchmarks indicate that the data generated by the Re$^{3}$Syn has longer dependencies and significantly enhances the model{'}s long-context capabilities. For reproducibility, we will release our codebase upon acceptance."
}