SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge
Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, Manfred Pinkal
Abstract
This report summarizes the results of the SemEval 2018 task on machine comprehension using commonsense knowledge. For this machine comprehension task, we created a new corpus, MCScript. It contains a high number of questions that require commonsense knowledge for finding the correct answer. 11 teams from 4 different countries participated in this shared task, most of them used neural approaches. The best performing system achieves an accuracy of 83.95%, outperforming the baselines by a large margin, but still far from the human upper bound, which was found to be at 98%.- Anthology ID:
- S18-1119
- Volume:
- Proceedings of the 12th International Workshop on Semantic Evaluation
- Month:
- June
- Year:
- 2018
- Address:
- New Orleans, Louisiana
- Editors:
- Marianna Apidianaki, Saif M. Mohammad, Jonathan May, Ekaterina Shutova, Steven Bethard, Marine Carpuat
- Venue:
- SemEval
- SIG:
- SIGLEX
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 747–757
- Language:
- URL:
- https://aclanthology.org/S18-1119
- DOI:
- 10.18653/v1/S18-1119
- Cite (ACL):
- Simon Ostermann, Michael Roth, Ashutosh Modi, Stefan Thater, and Manfred Pinkal. 2018. SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 747–757, New Orleans, Louisiana. Association for Computational Linguistics.
- Cite (Informal):
- SemEval-2018 Task 11: Machine Comprehension Using Commonsense Knowledge (Ostermann et al., SemEval 2018)
- PDF:
- https://preview.aclanthology.org/teach-a-man-to-fish/S18-1119.pdf
- Data
- ConceptNet, MCScript, MCTest, NewsQA, RACE, SQuAD, TriviaQA