Abstract
This paper presents work on novel machine translation (MT) systems between spoken and signed languages, where signed languages are represented in SignWriting, a sign language writing system. Our work seeks to address the lack of out-of-the-box support for signed languages in current MT systems and is based on the SignBank dataset, which contains pairs of spoken language text and SignWriting content. We introduce novel methods to parse, factorize, decode, and evaluate SignWriting, leveraging ideas from neural factored MT. In a bilingual setup—translating from American Sign Language to (American) English—our method achieves over 30 BLEU, while in two multilingual setups—translating in both directions between spoken languages and signed languages—we achieve over 20 BLEU. We find that common MT techniques used to improve spoken language translation similarly affect the performance of sign language translation. These findings validate our use of an intermediate text representation for signed languages to include them in natural language processing research.- Anthology ID:
- 2023.findings-eacl.127
- Volume:
- Findings of the Association for Computational Linguistics: EACL 2023
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Andreas Vlachos, Isabelle Augenstein
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1706–1724
- Language:
- URL:
- https://aclanthology.org/2023.findings-eacl.127
- DOI:
- 10.18653/v1/2023.findings-eacl.127
- Cite (ACL):
- Zifan Jiang, Amit Moryossef, Mathias Müller, and Sarah Ebling. 2023. Machine Translation between Spoken Languages and Signed Languages Represented in SignWriting. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1706–1724, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- Machine Translation between Spoken Languages and Signed Languages Represented in SignWriting (Jiang et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2023.findings-eacl.127.pdf