Open-Domain Sign Language Translation Learned from Online Video
Bowen Shi, Diane Brentari, Gregory Shakhnarovich, Karen Livescu
Abstract
Existing work on sign language translation – that is, translation from sign language videos into sentences in a written language – has focused mainly on (1) data collected in a controlled environment or (2) data in a specific domain, which limits the applicability to real-world settings. In this paper, we introduce OpenASL, a large-scale American Sign Language (ASL) - English dataset collected from online video sites (e.g., YouTube).OpenASL contains 288 hours of ASL videos in multiple domains from over 200 signers and is the largest publicly available ASL translation dataset to date. To tackle the challenges of sign language translation in realistic settings and without glosses, we propose a set of techniques including sign search as a pretext task for pre-training and fusion of mouthing and handshape features. The proposed techniques produce consistent and large improvements in translation quality, over baseline models basedon prior work.- Anthology ID:
- 2022.emnlp-main.427
- Volume:
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 6365–6379
- Language:
- URL:
- https://aclanthology.org/2022.emnlp-main.427
- DOI:
- 10.18653/v1/2022.emnlp-main.427
- Cite (ACL):
- Bowen Shi, Diane Brentari, Gregory Shakhnarovich, and Karen Livescu. 2022. Open-Domain Sign Language Translation Learned from Online Video. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6365–6379, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- Open-Domain Sign Language Translation Learned from Online Video (Shi et al., EMNLP 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2022.emnlp-main.427.pdf