Conversational Tutoring in VR Training: The Role of Game Context and State Variables

Maia Aguirre, Ariane Méndez, Aitor García-Pablos, Montse Cuadros, Arantza del Pozo, Oier Lopez de Lacalle, Ander Salaberria, Jeremy Barnes, Pablo Martínez, Muhammad Zeshan Afzal


Abstract
Virtual Reality (VR) training provides safe, cost-effective engagement with lifelike scenarios but lacks intuitive communication between users and the virtual environment. This study investigates the use of Large Language Models (LLMs) as conversational tutors in VR health and safety training, examining the impact of game context and state variables on LLM-generated answers in zero- and few-shot settings. Results demonstrate that incorporating both game context and state information significantly improves answer accuracy, with human evaluations showing gains of up to 0.26 points in zero-shot and 0.18 points in few-shot settings on a 0-1 scale.
Anthology ID:
2025.iwsds-1.23
Volume:
Proceedings of the 15th International Workshop on Spoken Dialogue Systems Technology
Month:
May
Year:
2025
Address:
Bilbao, Spain
Editors:
Maria Ines Torres, Yuki Matsuda, Zoraida Callejas, Arantza del Pozo, Luis Fernando D'Haro
Venues:
IWSDS | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
215–224
Language:
URL:
https://preview.aclanthology.org/fix-sig-urls/2025.iwsds-1.23/
DOI:
Bibkey:
Cite (ACL):
Maia Aguirre, Ariane Méndez, Aitor García-Pablos, Montse Cuadros, Arantza del Pozo, Oier Lopez de Lacalle, Ander Salaberria, Jeremy Barnes, Pablo Martínez, and Muhammad Zeshan Afzal. 2025. Conversational Tutoring in VR Training: The Role of Game Context and State Variables. In Proceedings of the 15th International Workshop on Spoken Dialogue Systems Technology, pages 215–224, Bilbao, Spain. Association for Computational Linguistics.
Cite (Informal):
Conversational Tutoring in VR Training: The Role of Game Context and State Variables (Aguirre et al., IWSDS 2025)
Copy Citation:
PDF:
https://preview.aclanthology.org/fix-sig-urls/2025.iwsds-1.23.pdf