@inproceedings{zarzu-zouhar-2026-hurdles,
    title = "Hurdles of Automatic Metric for Speech Translation Evaluation",
    author = "Zarzu, Victor Eugen  and
      Zouhar, Vilem",
    editor = "Salesky, Elizabeth  and
      Anastasopoulos, Antonios  and
      Negri, Matteo  and
      Federico, Marcello",
    booktitle = "Proceedings of the 23rd International Conference on Spoken Language Translation ({IWSLT} 2026)",
    month = jul,
    year = "2026",
    address = "San Diego, USA (in-person and online)",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/corrections-2026-06/2026.iwslt-1.34/",
    doi = "10.18653/v1/2026.iwslt-1.34",
    pages = "305--315",
    ISBN = "979-8-89176-411-8",
    abstract = "Automatic evaluation of speech translation has so far relied on text-only automated metrics that ignore speech phenomena. One would expect that incorporating the source audio modality would improve the performance of automatic metrics. We implement two standard metric paradigms: a COMET-audio regression model using audio and text encoders, and one based on prompting a speech large language model. Surprisingly, both audio-infused models fail to reliably surpass text-only baselines. We attribute this failure to the noise pollution and audio-transcript mismatches present in the audio signal, which makes the modality unreliable from the metric{'}s perspective. Furthermore, we argue that current human-annotated evaluation datasets for automated metrics predominantly feature technical content or short texts where paralinguistic features like prosody lack importance, rendering the extra audio information unhelpful for quality estimation (QE)."
}