Abstract
This paper concerns evaluating methods for extracting phonological information of Swedish Sign Language signs from video data with MediaPipe’s pose estimation. The methods involve estimating i) the articulation phase, ii) hand dominance (left vs. right), iii) the number of hands articulating (one- vs. two-handed signs) and iv) the sign’s place of articulation. The results show that MediaPipe’s tracking of the hands’ location and movement in videos can be used to estimate the articulation phase of signs. Whereas the inclusion of transport movements improves the accuracy for the estimation of hand dominance and number of hands, removing transport movements is crucial for estimating a sign’s place of articulation.- Anthology ID:
- 2023.nodalida-1.18
- Volume:
- Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)
- Month:
- May
- Year:
- 2023
- Address:
- Tórshavn, Faroe Islands
- Editors:
- Tanel Alumäe, Mark Fishel
- Venue:
- NoDaLiDa
- SIG:
- Publisher:
- University of Tartu Library
- Note:
- Pages:
- 169–178
- Language:
- URL:
- https://aclanthology.org/2023.nodalida-1.18
- DOI:
- Cite (ACL):
- Carl Börstell. 2023. Extracting Sign Language Articulation from Videos with MediaPipe. In Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pages 169–178, Tórshavn, Faroe Islands. University of Tartu Library.
- Cite (Informal):
- Extracting Sign Language Articulation from Videos with MediaPipe (Börstell, NoDaLiDa 2023)
- PDF:
- https://preview.aclanthology.org/emnlp22-frontmatter/2023.nodalida-1.18.pdf