Japanese-English Sentence Translation Exercises Dataset for Automatic Grading
Naoki Miura, Hiroaki Funayama, Seiya Kikuchi, Yuichiroh Matsubayashi, Yuya Iwase, Kentaro Inui
Abstract
This paper proposes the task of automatic assessment of Sentence Translation Exercises (STEs), that have been used in the early stage of L2 language learning.We formalize the task as grading student responses for each rubric criterion pre-specified by the educators.We then create a dataset for STE between Japanese and English including 21 questions, along with a total of 3,498 student responses (167 on average).The answer responses were collected from students and crowd workers.Using this dataset, we demonstrate the performance of baselines including a finetuned BERT model and GPT-3.5 with few-shot learning. Experimental results showed that the baseline model with fine-tuned BERT was able to classify correct responses with approximately 90% in F1, but only less than 80% for incorrect responses. Furthermore, GPT-3.5 with few-shot learning shows a poorer result than the BERT model, indicating that our newly proposed task presents a challenging issue, even for the state-of-the-art large language model.- Anthology ID:
- 2024.eacl-srw.21
- Volume:
- Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop
- Month:
- March
- Year:
- 2024
- Address:
- St. Julian’s, Malta
- Editors:
- Neele Falk, Sara Papi, Mike Zhang
- Venue:
- EACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 266–278
- Language:
- URL:
- https://aclanthology.org/2024.eacl-srw.21
- DOI:
- Cite (ACL):
- Naoki Miura, Hiroaki Funayama, Seiya Kikuchi, Yuichiroh Matsubayashi, Yuya Iwase, and Kentaro Inui. 2024. Japanese-English Sentence Translation Exercises Dataset for Automatic Grading. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 266–278, St. Julian’s, Malta. Association for Computational Linguistics.
- Cite (Informal):
- Japanese-English Sentence Translation Exercises Dataset for Automatic Grading (Miura et al., EACL 2024)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2024.eacl-srw.21.pdf