Action-Concentrated Embedding Framework: This Is Your Captain Sign-tokening

Hyunwook Yu, Suhyeon Shin, Junku Heo, Hyuntaek Shin, Hyosu Kim, Mucheol Kim

Abstract
Sign language is the primary communication medium for people who are deaf or have hearing loss. However, given the divergent range of sensory abilities of these individuals, there is a communication gap that needs to be addressed. In this paper, we present action-concentrated embedding (ACE), which is a novel sign token embedding framework. Additionally, to provide a more structured foundation for sign language analysis, we introduce a dedicated notation system tailored for sign language that endeavors to encapsulate the nuanced gestures and movements that are integral with sign communication. The proposed ACE approach tracks a signer’s actions based on human posture estimation. Tokenizing these actions and capturing the token embedding using a short-time Fourier transform encapsulates the time-based behavioral changes. Hence, ACE offers input embedding to translate sign language into natural language sentences. When tested against a disaster sign language dataset using automated machine translation measures, ACE notably surpasses prior research in terms of translation capabilities, improving the performance by up to 5.79% for BLEU-4 and 5.46% for ROUGE-L metric.
Anthology ID:
2024.lrec-main.29
Volume:
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, Nianwen Xue
Venues:
LREC | COLING
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
310–320
Language:
URL:
https://aclanthology.org/2024.lrec-main.29
DOI:
Bibkey:
Cite (ACL):
Hyunwook Yu, Suhyeon Shin, Junku Heo, Hyuntaek Shin, Hyosu Kim, and Mucheol Kim. 2024. Action-Concentrated Embedding Framework: This Is Your Captain Sign-tokening. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 310–320, Torino, Italia. ELRA and ICCL.
Cite (Informal):
Action-Concentrated Embedding Framework: This Is Your Captain Sign-tokening (Yu et al., LREC-COLING 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/landing_page/2024.lrec-main.29.pdf
PDF Search