@inproceedings{kim-etal-2026-aligning,
    title = "Aligning Paralinguistic Understanding and Generation in Speech {LLM}s via Multi-Task Reinforcement Learning",
    author = "Kim, Minseok  and
      Chen, Jingxiang  and
      Leem, Seong-Gyun  and
      Huang, Yin  and
      Rungta, Rashi  and
      Ouyang, Zhicheng  and
      Wu, Haibin  and
      Appini, Surya Teja  and
      Bansal, Ankur  and
      Bai, Yang  and
      Liu, Yue  and
      Metze, Florian  and
      Aly, Ahmed A  and
      Kumar, Anuj  and
      Rastrow, Ariya  and
      Lin, Zhaojiang",
    editor = {Matusevych, Yevgen  and
      Eryi{\u{g}}it, G{\"u}l{\c{s}}en  and
      Aletras, Nikolaos},
    booktitle = "Proceedings of the 19th Conference of the {E}uropean Chapter of the {A}ssociation for {C}omputational {L}inguistics (Volume 5: Industry Track)",
    month = mar,
    year = "2026",
    address = "Rabat, Morocco",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-eacl/2026.eacl-industry.49/",
    pages = "636--648",
    ISBN = "979-8-89176-384-5",
    abstract = "Speech large language models (LLMs) observe paralinguistic cues such as prosody, emotion, and non-verbal sounds{---}crucial for intent understanding. However, leveraging these cues faces challenges: limited training data, annotation difficulty, and models exploiting lexical shortcuts over paralinguistic signals. We propose multi-task reinforcement learning (RL) with chain-of-thought prompting that elicits explicit affective reasoning. To address data scarcity, we introduce a paralinguistics-aware speech LLM (PALLM) that jointly optimizes sentiment classification from audio and paralinguistics-aware response generation via a two-stage pipeline. Experiments demonstrate that our approach improves paralinguistics understanding over both supervised baselines and strong proprietary models (Gemini-2.5-Pro, GPT-4o-audio), by 8-12{\%} on Expresso, IEMOCAP, and RAVDESS. The results show that modeling paralinguistic reasoning with multi-task RL is crucial for building emotionally intelligent speech LLMs."
}