@inproceedings{yunfanxie-etal-2025-weak,
    title = "Weak-to-Strong Honesty Alignment via Learning-to-Rank Supervision",
    author = "YunfanXie, YunfanXie  and
      Zou, Lixin  and
      Luo, Dan  and
      Tang, Min  and
      Li, Chenliang",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/corrections-2025-08/2025.findings-acl.529/",
    doi = "10.18653/v1/2025.findings-acl.529",
    pages = "10154--10168",
    ISBN = "979-8-89176-256-5",
    abstract = "Honest alignment refers to the ability of a language model to truthfully convey its knowledge limitations by appropriately refusing to answer questions when it lacks sufficient information. Existing solutions, such as prompt engineering and fine-tuning, face limitations: the former provides only marginal improvements, while the latter struggles to enhance honesty when annotated data is scarce.To overcome the above limitations, we propose , a novel framework that enhances honesty through weak-to-strong generalization. Specifically, we train the strong LLMs under weak model supervision to improve their honesty. For the weak model, we employ a learning-to-rank strategy to train a ``honest head'', which learns to select the most honest response among model{'}s outputs generated through beam search. For the strong LLM, we leverage the self-labeled dataset to update its parameters. Our proposal requires only minimal training data to train the weak honest model, yet achieve decent performance for labeling data. In addition, it enables the strong LLMs to have the capabilities to generalize even facing with the flawed label data. Extensive experiments show significantly boosts honest alignment in large models even with limited labeled data. Our code is available at \url{https://github.com/zewanfaan/WHAT_Honesty}."
}