Towards the Application of Calibrated Transformers to the Unsupervised Estimation of Question Difficulty from Text
Ekaterina Loginova, Luca Benedetto, Dries Benoit, Paolo Cremonesi
Abstract
Being able to accurately perform Question Difficulty Estimation (QDE) can improve the accuracy of students’ assessment and better their learning experience. Traditional approaches to QDE are either subjective or introduce a long delay before new questions can be used to assess students. Thus, recent work proposed machine learning-based approaches to overcome these limitations. They use questions of known difficulty to train models capable of inferring the difficulty of questions from their text. Once trained, they can be used to perform QDE of newly created questions. Existing approaches employ supervised models which are domain-dependent and require a large dataset of questions of known difficulty for training. Therefore, they cannot be used if such a dataset is not available ( for new courses on an e-learning platform). In this work, we experiment with the possibility of performing QDE from text in an unsupervised manner. Specifically, we use the uncertainty of calibrated question answering models as a proxy of human-perceived difficulty. Our experiments show promising results, suggesting that model uncertainty could be successfully leveraged to perform QDE from text, reducing both costs and elapsed time.- Anthology ID:
- 2021.ranlp-1.97
- Volume:
- Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021)
- Month:
- September
- Year:
- 2021
- Address:
- Held Online
- Venue:
- RANLP
- SIG:
- Publisher:
- INCOMA Ltd.
- Note:
- Pages:
- 846–855
- Language:
- URL:
- https://aclanthology.org/2021.ranlp-1.97
- DOI:
- Cite (ACL):
- Ekaterina Loginova, Luca Benedetto, Dries Benoit, and Paolo Cremonesi. 2021. Towards the Application of Calibrated Transformers to the Unsupervised Estimation of Question Difficulty from Text. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 846–855, Held Online. INCOMA Ltd..
- Cite (Informal):
- Towards the Application of Calibrated Transformers to the Unsupervised Estimation of Question Difficulty from Text (Loginova et al., RANLP 2021)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2021.ranlp-1.97.pdf
- Data
- RACE