Abstract
In this paper, we consider the task of digitally voicing silent speech, where silently mouthed words are converted to audible speech based on electromyography (EMG) sensor measurements that capture muscle impulses. While prior work has focused on training speech synthesis models from EMG collected during vocalized speech, we are the first to train from EMG collected during silently articulated speech. We introduce a method of training on silent EMG by transferring audio targets from vocalized to silent signals. Our method greatly improves intelligibility of audio generated from silent EMG compared to a baseline that only trains with vocalized data, decreasing transcription word error rate from 64% to 4% in one data condition and 88% to 68% in another. To spur further development on this task, we share our new dataset of silent and vocalized facial EMG measurements.- Anthology ID:
- 2020.emnlp-main.445
- Volume:
- Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Editors:
- Bonnie Webber, Trevor Cohn, Yulan He, Yang Liu
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 5521–5530
- Language:
- URL:
- https://aclanthology.org/2020.emnlp-main.445
- DOI:
- 10.18653/v1/2020.emnlp-main.445
- Award:
- Best Paper
- Cite (ACL):
- David Gaddy and Dan Klein. 2020. Digital Voicing of Silent Speech. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5521–5530, Online. Association for Computational Linguistics.
- Cite (Informal):
- Digital Voicing of Silent Speech (Gaddy & Klein, EMNLP 2020)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2020.emnlp-main.445.pdf
- Code
- dgaddy/silent_speech
- Data
- Silent Speech EMG