@inproceedings{tan-etal-2025-seeing,
    title = "Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generation",
    author = "Tan, Weiting  and
      Lian, Jiachen  and
      Inaguma, Hirofumi  and
      Tomasello, Paden  and
      Koehn, Philipp  and
      Ma, Xutai",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.140/",
    doi = "10.18653/v1/2025.findings-emnlp.140",
    pages = "2600--2617",
    ISBN = "979-8-89176-335-7",
    abstract = "We present an Audio-Visual Language Model (AVLM) for expressive speech generation by integrating full-face visual cues into a pre-trained expressive speech model. We explore multiple visual encoders and multimodal fusion strategies during pre-training to identify the most effective integration approach. Subsequent fine-tuning on emotion recognition and expressive dialogue tasks yields substantial gains over speech-only baselines (e.g., $+5$ F1 in emotion recognition). AVLM highlights the value of expressive visual information in guiding speech generation and offers a foundation for end-to-end multimodal conversational systems."
}