The Key Points: Using Feature Importance to Identify Shortcomings in Sign Language Recognition Models

Ruth M. Holmes, Ellen Rushe, Anthony Ventresque

Abstract
Pose estimation keypoints are widely used in sign language recognition (SLR) as a means of generalising to unseen signers. Despite the advantages of keypoints, SLR models struggle to achieve high recognition accuracy for many signed languages due to the large degree of variability between occurrences of the same signs, the lack of large datasets and the imbalanced nature of the data therein. In this paper we seek to provide a deeper analysis into the ways that these keypoints are used by models in order to determine which are most informative to SLR, identify potentially redundant ones and investigate whether keypoints that are central to differentiating signs in practice are being effectively used as expected by models.
Anthology ID:
2024.lrec-main.1387
Volume:
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, Nianwen Xue
Venues:
LREC | COLING
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
15970–15975
Language:
URL:
https://aclanthology.org/2024.lrec-main.1387
DOI:
Bibkey:
Cite (ACL):
Ruth M. Holmes, Ellen Rushe, and Anthony Ventresque. 2024. The Key Points: Using Feature Importance to Identify Shortcomings in Sign Language Recognition Models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 15970–15975, Torino, Italia. ELRA and ICCL.
Cite (Informal):
The Key Points: Using Feature Importance to Identify Shortcomings in Sign Language Recognition Models (Holmes et al., LREC-COLING 2024)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2024.lrec-main.1387.pdf
PDF Search