An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation
Amit Moryossef, Mathias Müller, Anne Göhring, Zifan Jiang, Yoav Goldberg, Sarah Ebling
Abstract
Sign language translation systems are complex and require many components. As a result, it is very hard to compare methods across publications. We present an open-source implementation of a text-to-gloss-to-pose-to-video pipeline approach, demonstrating conversion from German to Swiss German Sign Language, French to French Sign Language of Switzerland, and Italian to Italian Sign Language of Switzerland. We propose three different components for the text-to-gloss translation: a lemmatizer, a rule-based word reordering and dropping component, and a neural machine translation system. Gloss-to-pose conversion occurs using data from a lexicon for three different signed languages, with skeletal poses extracted from videos. To generate a sentence, the text-to-gloss system is first run, and the pose representations of the resulting signs are stitched together.- Anthology ID:
- 2023.at4ssl-1.3
- Volume:
- Proceedings of the Second International Workshop on Automatic Translation for Signed and Spoken Languages
- Month:
- June
- Year:
- 2023
- Address:
- Tampere, Finland
- Editors:
- Dimitar Shterionov, Mirella De Sisto, Mathias Muller, Davy Van Landuyt, Rehana Omardeen, Shaun Oboyle, Annelies Braffort, Floris Roelofsen, Fred Blain, Bram Vanroy, Eleftherios Avramidis
- Venue:
- AT4SSL
- SIG:
- Publisher:
- European Association for Machine Translation
- Note:
- Pages:
- 22–33
- Language:
- URL:
- https://preview.aclanthology.org/icon-24-ingestion/2023.at4ssl-1.3/
- DOI:
- Cite (ACL):
- Amit Moryossef, Mathias Müller, Anne Göhring, Zifan Jiang, Yoav Goldberg, and Sarah Ebling. 2023. An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation. In Proceedings of the Second International Workshop on Automatic Translation for Signed and Spoken Languages, pages 22–33, Tampere, Finland. European Association for Machine Translation.
- Cite (Informal):
- An Open-Source Gloss-Based Baseline for Spoken to Signed Language Translation (Moryossef et al., AT4SSL 2023)
- PDF:
- https://preview.aclanthology.org/icon-24-ingestion/2023.at4ssl-1.3.pdf