Predicting Algorithm Classes for Programming Word Problems

Vinayak Athavale, Aayush Naik, Rajas Vanjape, Manish Shrivastava

Abstract
We introduce the task of algorithm class prediction for programming word problems. A programming word problem is a problem written in natural language, which can be solved using an algorithm or a program. We define classes of various programming word problems which correspond to the class of algorithms required to solve the problem. We present four new datasets for this task, two multiclass datasets with 550 and 1159 problems each and two multilabel datasets having 3737 and 3960 problems each. We pose the problem as a text classification problem and train neural network and non-neural network based models on this task. Our best performing classifier gets an accuracy of 62.7 percent for the multiclass case on the five class classification dataset, Codeforces Multiclass-5 (CFMC5). We also do some human-level analysis and compare human performance with that of our text classification models. Our best classifier has an accuracy only 9 percent lower than that of a human on this task. To the best of our knowledge, these are the first reported results on such a task. We make our code and datasets publicly available.
Anthology ID:
D19-5511
Volume:
Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)
Month:
November
Year:
2019
Address:
Hong Kong, China
Editors:
Wei Xu, Alan Ritter, Tim Baldwin, Afshin Rahimi
Venue:
WNUT
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
84–93
Language:
URL:
https://preview.aclanthology.org/remove-affiliations/D19-5511/
DOI:
10.18653/v1/D19-5511
Bibkey:
Cite (ACL):
Vinayak Athavale, Aayush Naik, Rajas Vanjape, and Manish Shrivastava. 2019. Predicting Algorithm Classes for Programming Word Problems. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 84–93, Hong Kong, China. Association for Computational Linguistics.
Cite (Informal):
Predicting Algorithm Classes for Programming Word Problems (Athavale et al., WNUT 2019)
Copy Citation:
PDF:
https://preview.aclanthology.org/remove-affiliations/D19-5511.pdf
PDF Search Fix data