@inproceedings{wang-etal-2025-audiobench,
    title = "{A}udio{B}ench: A Universal Benchmark for Audio Large Language Models",
    author = "Wang, Bin  and
      Zou, Xunlong  and
      Lin, Geyu  and
      Sun, Shuo  and
      Liu, Zhuohan  and
      Zhang, Wenyu  and
      Liu, Zhengyuan  and
      Aw, AiTi  and
      Chen, Nancy F.",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.naacl-long.218/",
    pages = "4297--4316",
    ISBN = "979-8-89176-189-6",
    abstract = "We introduce AudioBench, a universal benchmark designed to evaluate Audio Large Language Models (AudioLLMs). It encompasses 8 distinct tasks and 26 datasets, among which, 7 are newly proposed datasets. The evaluation targets three main aspects: speech understanding, audio scene understanding, and voice understanding (paralinguistic). Despite recent advancements, there lacks a comprehensive benchmark for AudioLLMs on instruction following capabilities conditioned on audio signals. AudioBench addresses this gap by setting up datasets as well as desired evaluation metrics. Besides, we also evaluated the capabilities of five popular models and found that no single model excels consistently across all tasks. We outline the research outlook for AudioLLMs and anticipate that our open-sourced evaluation toolkit, data, and leaderboard will offer a robust testbed for future model developments."
}