@inproceedings{cho-etal-2024-speechworthy,
    title = "Speechworthy Instruction-tuned Language Models",
    author = "Cho, Hyundong Justin  and
      Jedema, Nicolaas Paul  and
      Ribeiro, Leonardo F. R.  and
      Sharma, Karishma  and
      Szekely, Pedro  and
      Moschitti, Alessandro  and
      Janssen, Ruben  and
      May, Jonathan",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.emnlp-main.595/",
    doi = "10.18653/v1/2024.emnlp-main.595",
    pages = "10652--10670",
    abstract = "Current instruction-tuned language models are exclusively trained with textual preference data and thus may not be aligned to the unique requirements of other modalities, such as speech. To better align language models with the speech domain, we explore i) prompting strategies based on radio-industry best practices and ii) preference learning using a novel speech-based preference data of 20K samples collected by annotators who listen to response pairs. Both human and automatic evaluation show that both prompting and preference learning increase the speech-suitability of popular instruction tuned LLMs. More interestingly, we show that these methods are additive; combining them achieves the best win rates in head-to-head comparison, resulting in responses that are preferred or tied to the base model in 76.2{\%} of comparisons on average. Lastly, we share lexical, syntactical, and qualitative analyses that elicit how our studied methods differ with baselines in generating more speech-suitable responses."
}