On the application of Transformers for estimating the difficulty of Multiple-Choice Questions from text

Luca Benedetto, Giovanni Aradelli, Paolo Cremonesi, Andrea Cappelli, Andrea Giussani, Roberto Turrin

Abstract
Classical approaches to question calibration are either subjective or require newly created questions to be deployed before being calibrated. Recent works explored the possibility of estimating question difficulty from text, but did not experiment with the most recent NLP models, in particular Transformers. In this paper, we compare the performance of previous literature with Transformer models experimenting on a public and a private dataset. Our experimental results show that Transformers are capable of outperforming previously proposed models. Moreover, if an additional corpus of related documents is available, Transformers can leverage that information to further improve calibration accuracy. We characterize the dependence of the model performance on some properties of the questions, showing that it performs best on questions ending with a question mark and Multiple-Choice Questions (MCQs) with one correct choice.
Anthology ID:
2021.bea-1.16
Volume:
Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications
Month:
April
Year:
2021
Address:
Online
Editors:
Jill Burstein, Andrea Horbach, Ekaterina Kochmar, Ronja Laarmann-Quante, Claudia Leacock, Nitin Madnani, Ildikó Pilán, Helen Yannakoudakis, Torsten Zesch
Venue:
BEA
SIG:
SIGEDU
Publisher:
Association for Computational Linguistics
Note:
Pages:
147–157
Language:
URL:
https://aclanthology.org/2021.bea-1.16
DOI:
Bibkey:
Cite (ACL):
Luca Benedetto, Giovanni Aradelli, Paolo Cremonesi, Andrea Cappelli, Andrea Giussani, and Roberto Turrin. 2021. On the application of Transformers for estimating the difficulty of Multiple-Choice Questions from text. In Proceedings of the 16th Workshop on Innovative Use of NLP for Building Educational Applications, pages 147–157, Online. Association for Computational Linguistics.
Cite (Informal):
On the application of Transformers for estimating the difficulty of Multiple-Choice Questions from text (Benedetto et al., BEA 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-4/2021.bea-1.16.pdf
PDF Search