FBK@IWSLT Test Suites Task: Gender Bias evaluation with MuST-SHE
Beatrice Savoldi, Marco Gaido, Matteo Negri, Luisa Bentivogli
Abstract
This paper presents the FBK contribution to the IWSLT-2024 ‘Test suites’ shared subtask, part of the Offline Speech Translation Task. Our contribution consists of the MuST-SHE-IWSLT24 benchmark evaluation, designed to assess gender bias in speech translation. By focusing on the en-de language pair, we rely on a newly created test suite to investigate systems’ ability to correctly translate feminine and masculine gender. Our results indicate that – under realistic conditions – current ST systems achieve reasonable and comparable performance in correctly translating both feminine and masculine forms when contextual gender information is available. For ambiguous references to the speaker, however, we attest a consistent preference towards masculine gender, thus calling for future endeavours on the topic. Towards this goal we make MuST-SHE-IWSLT24 freely available at: https://mt.fbk.eu/must-she/- Anthology ID:
- 2024.iwslt-1.10
- Volume:
- Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024)
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand (in-person and online)
- Editors:
- Elizabeth Salesky, Marcello Federico, Marine Carpuat
- Venue:
- IWSLT
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 65–71
- Language:
- URL:
- https://aclanthology.org/2024.iwslt-1.10
- DOI:
- 10.18653/v1/2024.iwslt-1.10
- Cite (ACL):
- Beatrice Savoldi, Marco Gaido, Matteo Negri, and Luisa Bentivogli. 2024. FBK@IWSLT Test Suites Task: Gender Bias evaluation with MuST-SHE. In Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024), pages 65–71, Bangkok, Thailand (in-person and online). Association for Computational Linguistics.
- Cite (Informal):
- FBK@IWSLT Test Suites Task: Gender Bias evaluation with MuST-SHE (Savoldi et al., IWSLT 2024)
- PDF:
- https://preview.aclanthology.org/autopr/2024.iwslt-1.10.pdf