Abstract
Sign language is an effective non-verbal communication mode for the hearing-impaired people. Since the video-based sign language detection models have high requirements for enough lighting and clear background, current wearing glove-based sign language models are robust for poor light and occlusion situations. In this paper, we annotate a new dataset of Word-based Wearable Chinese Sign Languag (WW-CSL) gestures. Specifically, we propose a three-form (e.g., sequential sensor data, gesture video, and gesture text) scheme to represent dynamic CSL gestures. Guided by the scheme, a total of 3,000 samples were collected, corresponding to 100 word-based CSL gestures. Furthermore, we present a transformer-based baseline model to fuse 2 inertial measurement unites (IMUs) and 10 flex sensors for the wearable CSL detection. In order to integrate the advantage of video-based and wearable glove-based CSL gestures, we also propose a transformer-based Multi-Modal CSL Detection (MM-CSLD) framework which adeptly integrates the local sequential sensor data derived from wearable-based CSL gestures with the global, fine-grained skeleton representations captured from video-based CSL gestures simultaneously.- Anthology ID:
- 2024.lrec-main.1541
- Volume:
- Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
- Month:
- May
- Year:
- 2024
- Address:
- Torino, Italia
- Editors:
- Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, Nianwen Xue
- Venues:
- LREC | COLING
- SIG:
- Publisher:
- ELRA and ICCL
- Note:
- Pages:
- 17718–17724
- Language:
- URL:
- https://preview.aclanthology.org/add_missing_videos/2024.lrec-main.1541/
- DOI:
- Cite (ACL):
- Fan Xu, Kai Liu, Yifeng Yang, and Keyu Yan. 2024. WW-CSL: A New Dataset for Word-Based Wearable Chinese Sign Language Detection. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 17718–17724, Torino, Italia. ELRA and ICCL.
- Cite (Informal):
- WW-CSL: A New Dataset for Word-Based Wearable Chinese Sign Language Detection (Xu et al., LREC-COLING 2024)
- PDF:
- https://preview.aclanthology.org/add_missing_videos/2024.lrec-main.1541.pdf