@inproceedings{feng-etal-2025-sequence,
    title = "Sequence-level Large Language Model Training with Contrastive Preference Optimization",
    author = "Feng, Zhili  and
      Ram, Dhananjay  and
      Hawkins, Cole  and
      Rawal, Aditya  and
      Zhao, Jinman  and
      Zha, Sheng",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.findings-naacl.233/",
    pages = "4158--4164",
    ISBN = "979-8-89176-195-7",
    abstract = "The next token prediction loss is the dominant self-supervised training objective for large language models and has achieved promising results in a variety of downstream tasks. However, upon closer investigation of this objective, we find that it lacks an understanding of sequence-level signals, leading to a mismatch between training and inference processes. To bridge this gap, we introduce a contrastive preference optimization (CPO) procedure that can inject sequence-level information into the language model at any training stage without expensive human labeled data. Our experiments show that the proposed objective surpasses the next token prediction in terms of win rate in the instruction-following and text generation tasks."
}