@inproceedings{herron-etal-2026-responsible,
    title = "Responsible Benchmarking of Fairness for Automatic Speech Recognition",
    author = "Herron, Felix E.  and
      Richard, Ange  and
      Portet, Fran{\c{c}}ois  and
      Allauzen, Alexandre  and
      Rossato, Solange",
    editor = "Hosseini-Kivanani, Nina  and
      Brutti, Alessio  and
      Matassoni, Marco  and
      Dowerah, Sandipana  and
      Liga, Davide  and
      Schommer, Christoph",
    booktitle = "Proceedings of Speech Language Models in Low-Resource Settings: Performance, Evaluation, and Bias Analysis ({SPEAKABLE}) @ {LREC} 2026",
    month = may,
    year = "2026",
    address = "Palma, Mallorca (Spain)",
    publisher = "ELRA Language Resources Association (ELRA)",
    url = "https://preview.aclanthology.org/backfill-openreview/2026.speakable-1.8/",
    doi = "10.63317/3jpc2uj4pp3g",
    pages = "66--78",
    abstract = "Many studies have shown automatic speech processing (ASR) systems have unequal performance across speaker groups (SG{'}s). However, the manner in which such studies arrive at this conclusion is inconsistent. To pave the way for more reliable results in future studies, we lay out best practices for benchmarking ASR fairness based on literature from machine learning fairness, social sciences, and speech science. We then perform a case study on the Fair-speech benchmark, applying aforementioned best practices, and discuss how failing to do so can result in erroneous conclusions. On the whole, we advocate for as fine-grained an analysis as possible, taking into account as many variables as are available, in order to eschew dataset-level bias."
}