Bidirectional Skeleton-Based Isolated Sign Recognition using Graph Convolutional Networks

Konstantinos M. Dafnis, Evgenia Chroni, Carol Neidle, Dimitri Metaxas

Abstract
To improve computer-based recognition from video of isolated signs from American Sign Language (ASL), we propose a new skeleton-based method that involves explicit detection of the start and end frames of signs, trained on the ASLLVD dataset; it uses linguistically relevant parameters based on the skeleton input. Our method employs a bidirectional learning approach within a Graph Convolutional Network (GCN) framework. We apply this method to the WLASL dataset, but with corrections to the gloss labeling to ensure consistency in the labels assigned to different signs; it is important to have a 1-1 correspondence between signs and text-based gloss labels. We achieve a success rate of 77.43% for top-1 and 94.54% for top-5 using this modified WLASL dataset. Our method, which does not require multi-modal data input, outperforms other state-of-the-art approaches on the same modified WLASL dataset, demonstrating the importance of both attention to the start and end frames of signs and the use of bidirectional data streams in the GCNs for isolated sign recognition.
Anthology ID:
2022.lrec-1.797
Volume:
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Month:
June
Year:
2022
Address:
Marseille, France
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
7328–7338
Language:
URL:
https://aclanthology.org/2022.lrec-1.797
DOI:
Bibkey:
Cite (ACL):
Konstantinos M. Dafnis, Evgenia Chroni, Carol Neidle, and Dimitri Metaxas. 2022. Bidirectional Skeleton-Based Isolated Sign Recognition using Graph Convolutional Networks. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 7328–7338, Marseille, France. European Language Resources Association.
Cite (Informal):
Bidirectional Skeleton-Based Isolated Sign Recognition using Graph Convolutional Networks (Dafnis et al., LREC 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2022.lrec-1.797.pdf
Data
AUTSL
PDF Search