@inproceedings{garces-arias-etal-2024-adaptive,
    title = "Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation",
    author = "Garces Arias, Esteban  and
      Rodemann, Julian  and
      Li, Meimingwei  and
      Heumann, Christian  and
      A{\ss}enmacher, Matthias",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2024",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.findings-emnlp.885/",
    doi = "10.18653/v1/2024.findings-emnlp.885",
    pages = "15060--15080",
    abstract = "Despite the remarkable capabilities of large language models, generating high-quality text remains a challenging task. Numerous decoding strategies{---}such as beam search, sampling with temperature, top{-}$k$ sampling, nucleus (top{-}$p$) sampling, typical decoding, contrastive decoding, and contrastive search{---}have been proposed to address these challenges by improving coherence, diversity, and resemblance to human-generated text. In this study, we introduce Adaptive Contrastive Search (ACS), a novel decoding strategy that extends contrastive search (CS) by incorporating an adaptive degeneration penalty informed by the model{'}s estimated uncertainty at each generation step. ACS aims to enhance creativity and diversity while maintaining coherence to produce high-quality outputs. Extensive experiments across various model architectures, languages, and datasets demonstrate that our approach improves both creativity and coherence, underscoring its effectiveness in text-generation tasks. We release our code, datasets, and models to facilitate further research."
}