Abstract
The multi-modality nature of human communication has been utilized to enhance the performance of language modeling-related tasks. Driven by the development of large-scale end-to-end learning techniques and the availability of multi-modal data, it becomes possible to represent non-verbal communication behaviors through joint-learning, and directly study their interaction with verbal communication. However, there is still gaps in existing studies to better address the underlying mechanism of how non-verbal expression contributes to the overall communication purpose. Therefore, we explore two questions using mixed-modal language models trained against monologue video data: first, whether incorporating gesture representations can improve the language model’s performance (perplexity); second, whether spontaneous gestures demonstrate entropy rate constancy (ERC), which is an empirical pattern found in most verbal language data that supports the rational communication assumption from Information Theory. We have positive and interesting findings for both questions: speakers indeed use spontaneous gestures to convey “meaningful” information that enhances verbal communication, which can be captured with a simple spatial encoding scheme. More importantly, gestures are produced and organized rationally in a similar way as words, which optimizes the communication efficiency.- Anthology ID:
- 2023.findings-acl.600
- Volume:
- Findings of the Association for Computational Linguistics: ACL 2023
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Anna Rogers, Jordan Boyd-Graber, Naoaki Okazaki
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 9409–9424
- Language:
- URL:
- https://aclanthology.org/2023.findings-acl.600
- DOI:
- 10.18653/v1/2023.findings-acl.600
- Cite (ACL):
- Yang Xu and Yang Cheng. 2023. Spontaneous gestures encoded by hand positions improve language models: An Information-Theoretic motivated study. In Findings of the Association for Computational Linguistics: ACL 2023, pages 9409–9424, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- Spontaneous gestures encoded by hand positions improve language models: An Information-Theoretic motivated study (Xu & Cheng, Findings 2023)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2023.findings-acl.600.pdf