A Dataset and Benchmark on Extraction of Novel Concepts on Trust in AI from Scientific Literature

Melanie McGrath, Harrison Bailey, Necva Bölücü, Xiang Dai, Sarvnaz Karimi, Andreas Duenser, Cécile Paris


Abstract
This study investigates the extent to which linguistic typology influences the performance of two automatic speech recognition (ASR) systems across diverse language families. Using the FLEURS corpus and typological features from the World Atlas of Language Structures (WALS), we analysed 40 languages grouped by phonological, morphological, syntactic, and semantic domains. We evaluated two state-of-the-art multilingual ASR systems, Whisper and Seamless, to examine how their performance, measured by word error rate (WER), correlates with linguistic structures. Random Forests and Mixed Effects Models were used to quantify feature impact and statistical significance. Results reveal that while both systems leverage typological patterns, they differ in their sensitivity to specific domains. Our findings highlight how structural and functional linguistic features shape ASR performance, offering insights into model generalisability and typology-aware system development.
Anthology ID:
2025.alta-main.11
Volume:
Proceedings of The 23rd Annual Workshop of the Australasian Language Technology Association
Month:
November
Year:
2025
Address:
Sydney, Australia
Editors:
Jonathan K. Kummerfeld, Aditya Joshi, Mark Dras
Venue:
ALTA
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
159–175
Language:
URL:
https://preview.aclanthology.org/ingest-alta/2025.alta-main.11/
DOI:
Bibkey:
Cite (ACL):
Melanie McGrath, Harrison Bailey, Necva Bölücü, Xiang Dai, Sarvnaz Karimi, Andreas Duenser, and Cécile Paris. 2025. A Dataset and Benchmark on Extraction of Novel Concepts on Trust in AI from Scientific Literature. In Proceedings of The 23rd Annual Workshop of the Australasian Language Technology Association, pages 159–175, Sydney, Australia. Association for Computational Linguistics.
Cite (Informal):
A Dataset and Benchmark on Extraction of Novel Concepts on Trust in AI from Scientific Literature (McGrath et al., ALTA 2025)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-alta/2025.alta-main.11.pdf