Personalized Multimodal Feedback Generation in Education

Haochen Liu, Zitao Liu, Zhongqin Wu, Jiliang Tang

Abstract
The automatic feedback of school assignments is an important application of AI in education. In this work, we focus on the task of personalized multimodal feedback generation, which aims to generate personalized feedback for teachers to evaluate students’ assignments involving multimodal inputs such as images, audios, and texts. This task involves the representation and fusion of multimodal information and natural language generation, which presents the challenges from three aspects: (1) how to encode and integrate multimodal inputs; (2) how to generate feedback specific to each modality; and (3) how to fulfill personalized feedback generation. In this paper, we propose a novel Personalized Multimodal Feedback Generation Network (PMFGN) armed with a modality gate mechanism and a personalized bias mechanism to address these challenges. Extensive experiments on real-world K-12 education data show that our model significantly outperforms baselines by generating more accurate and diverse feedback. In addition, detailed ablation experiments are conducted to deepen our understanding of the proposed framework.
Anthology ID:
2020.coling-main.166
Volume:
Proceedings of the 28th International Conference on Computational Linguistics
Month:
December
Year:
2020
Address:
Barcelona, Spain (Online)
Venue:
COLING
SIG:
Publisher:
International Committee on Computational Linguistics
Note:
Pages:
1826–1840
Language:
URL:
https://aclanthology.org/2020.coling-main.166
DOI:
10.18653/v1/2020.coling-main.166
Bibkey:
Cite (ACL):
Haochen Liu, Zitao Liu, Zhongqin Wu, and Jiliang Tang. 2020. Personalized Multimodal Feedback Generation in Education. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1826–1840, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Cite (Informal):
Personalized Multimodal Feedback Generation in Education (Liu et al., COLING 2020)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2020.coling-main.166.pdf
PDF Search