@inproceedings{li-etal-2025-token,
    title = "Token-level Preference Self-Alignment Optimization for Multi-style Outline Controllable Generation",
    author = "Li, Zihao  and
      Xu, Xuekong  and
      Chen, Ziyao  and
      Zou, Lixin  and
      Ethanhjwu, Ethanhjwu  and
      Chen, Qiang  and
      Li, Chenliang",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.findings-acl.823/",
    pages = "15974--16007",
    ISBN = "979-8-89176-256-5",
    abstract = "Multi-style outline controllable generation is crucial for multiple applications, including document semantic structuring and retrieval-augmented generation.The great success of preference alignment approaches encourages their application in controllable generation tasks.However, these attempts encounter several limitations: (1) response pair requirements, (2) substantial computation costs, and (3) insufficient exploitation of fine-grained preference signals.To address these problems, we propose a token-level preference self-alignment optimization, named TKPO, for outline controllable generation. TKPO extends the Bradley-Terry model from pair-wise to list-wise comparison, which is further applied at the token level for fine-grained preference signal utilization. In comparison to the representative methods, e.g., DPO, TKPO does not require response pairs; instead, we propose a controllable attributes-driven method to construct reject samples for self-alignment. Additionally, TKPO optimizes only the base model, thereby avoiding additional memory usage and substantial computational costs.We curate two outline controllable generation datasets with regard to language style and level-of-detail.Extensive experiments demonstrate that TKPO outperforms DPO by up to 19.28{\%} in performance while requiring only 56.25{\%} in training time.We release the code and datasets resources at https://github.com/WHUIR/TKPO."
}