@inproceedings{yang-etal-2024-preference,
    title = "Not All Preference Pairs Are Created Equal: A Recipe for Annotation-Efficient Iterative Preference Learning",
    author = "Yang, Sen  and
      Cui, Leyang  and
      Cai, Deng  and
      Huang, Xinting  and
      Shi, Shuming  and
      Lam, Wai",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2024.findings-emnlp.382/",
    doi = "10.18653/v1/2024.findings-emnlp.382",
    pages = "6549--6561",
    abstract = "Iterative preference learning, though yielding superior performances, requires online annotated preference labels. In this work, we study strategies to save annotation budgets while achieving competitive or even better performances for iterative preference learning. Built on intuitions from active learning, we empirically show that annotating those response pairs with $small$ margins is generally better than $large$ or $random$. Besides, experiments under the multi-iteration scenario suggest allocating more annotation budgets in the earlier iterations rather than later ones."
}