InteLLA: Intelligent Language Learning Assistant for Assessing Language Proficiency through Interviews and Roleplays
Mao Saeki, Hiroaki Takatsu, Fuma Kurata, Shungo Suzuki, Masaki Eguchi, Ryuki Matsuura, Kotaro Takizawa, Sadahiro Yoshikawa, Yoichi Matsuyama
Abstract
In this paper, we propose a multimodal dialogue system designed to elicit spontaneous speech samples from second language learners for reliable oral proficiency assessment. The primary challenge in utilizing dialogue systems for language testing lies in obtaining ratable speech samples that demonstrates the user’s full capabilities of interactional skill. To address this, we developed a virtual agent capable of conducting extended interactions, consisting of a 15-minute interview and 10-minute roleplay. The interview component is a system-led dialogue featuring questions that aim to elicit specific language functions from the user. The system dynamically adjusts the topic difficulty based on real-time assessments to provoke linguistic breakdowns as evidence of their upper limit of proficiency. The roleplay component is a mixed-initiative, collaborative conversation aimed at evaluating the user’s interactional competence. Two experiments were conducted to evaluate our system’s reliability in assessing oral proficiency. In experiment 1, we collected a total of 340 interview sessions, 45-72% of which successfully elicited upper linguistic limit by adjusting the topic difficulty levels. In experiment 2, based on the ropleplay dataset of 75 speakers, the interactional speech elicited by our system was found to be as ratable as those by human examiners, indicated by the reliability index of interactional ratings. These results demonstrates that our system can elicit ratable interactional performances comparable to those elicited by human interviewers. Finally, we report on the deployment of our system with over 10,000 university students in a real-world testing scenario.- Anthology ID:
- 2024.sigdial-1.34
- Volume:
- Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue
- Month:
- September
- Year:
- 2024
- Address:
- Kyoto, Japan
- Editors:
- Tatsuya Kawahara, Vera Demberg, Stefan Ultes, Koji Inoue, Shikib Mehri, David Howcroft, Kazunori Komatani
- Venue:
- SIGDIAL
- SIG:
- SIGDIAL
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 385–399
- Language:
- URL:
- https://aclanthology.org/2024.sigdial-1.34
- DOI:
- 10.18653/v1/2024.sigdial-1.34
- Cite (ACL):
- Mao Saeki, Hiroaki Takatsu, Fuma Kurata, Shungo Suzuki, Masaki Eguchi, Ryuki Matsuura, Kotaro Takizawa, Sadahiro Yoshikawa, and Yoichi Matsuyama. 2024. InteLLA: Intelligent Language Learning Assistant for Assessing Language Proficiency through Interviews and Roleplays. In Proceedings of the 25th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 385–399, Kyoto, Japan. Association for Computational Linguistics.
- Cite (Informal):
- InteLLA: Intelligent Language Learning Assistant for Assessing Language Proficiency through Interviews and Roleplays (Saeki et al., SIGDIAL 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2024.sigdial-1.34.pdf