Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformers
Hosein Mohebbi, Grzegorz Chrupała, Willem Zuidema, Afra Alishahi
Abstract
Transformers have become a key architecture in speech processing, but our understanding of how they build up representations of acoustic and linguistic structure is limited. In this study, we address this gap by investigating how measures of ‘context-mixing’ developed for text models can be adapted and applied to models of spoken language. We identify a linguistic phenomenon that is ideal for such a case study: homophony in French (e.g. livre vs livres), where a speech recognition model has to attend to syntactic cues such as determiners and pronouns in order to disambiguate spoken words with identical pronunciations and transcribe them while respecting grammatical agreement. We perform a series of controlled experiments and probing analyses on Transformer-based speech models. Our findings reveal that representations in encoder-only models effectively incorporate these cues to identify the correct transcription, whereas encoders in encoder-decoder models mainly relegate the task of capturing contextual dependencies to decoder modules.- Anthology ID:
- 2023.emnlp-main.513
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 8249–8260
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-main.513
- DOI:
- 10.18653/v1/2023.emnlp-main.513
- Cite (ACL):
- Hosein Mohebbi, Grzegorz Chrupała, Willem Zuidema, and Afra Alishahi. 2023. Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformers. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8249–8260, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Homophone Disambiguation Reveals Patterns of Context Mixing in Speech Transformers (Mohebbi et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2023.emnlp-main.513.pdf