Abstract
Developing plausible distractors (wrong answer options) when writing multiple-choice questions has been described as one of the most challenging and time-consuming parts of the item-writing process. In this paper we propose a fully automatic method for generating distractor suggestions for multiple-choice questions used in high-stakes medical exams. The system uses a question stem and the correct answer as an input and produces a list of suggested distractors ranked based on their similarity to the stem and the correct answer. To do this we use a novel approach of combining concept embeddings with information retrieval methods. We frame the evaluation as a prediction task where we aim to “predict” the human-produced distractors used in large sets of medical questions, i.e. if a distractor generated by our system is good enough it is likely to feature among the list of distractors produced by the human item-writers. The results reveal that combining concept embeddings with information retrieval approaches significantly improves the generation of plausible distractors and enables us to match around 1 in 5 of the human-produced distractors. The approach proposed in this paper is generalisable to all scenarios where the distractors refer to concepts.- Anthology ID:
- W18-0548
- Volume:
- Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications
- Month:
- June
- Year:
- 2018
- Address:
- New Orleans, Louisiana
- Editors:
- Joel Tetreault, Jill Burstein, Ekaterina Kochmar, Claudia Leacock, Helen Yannakoudakis
- Venue:
- BEA
- SIG:
- SIGEDU
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 389–398
- Language:
- URL:
- https://aclanthology.org/W18-0548
- DOI:
- 10.18653/v1/W18-0548
- Cite (ACL):
- Le An Ha and Victoria Yaneva. 2018. Automatic Distractor Suggestion for Multiple-Choice Tests Using Concept Embeddings and Information Retrieval. In Proceedings of the Thirteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 389–398, New Orleans, Louisiana. Association for Computational Linguistics.
- Cite (Informal):
- Automatic Distractor Suggestion for Multiple-Choice Tests Using Concept Embeddings and Information Retrieval (Ha & Yaneva, BEA 2018)
- PDF:
- https://preview.aclanthology.org/emnlp22-frontmatter/W18-0548.pdf