@inproceedings{babe-etal-2024-studenteval,
    title = "{S}tudent{E}val: A Benchmark of Student-Written Prompts for Large Language Models of Code",
    author = "Babe, Hannah McLean  and
      Nguyen, Sydney  and
      Zi, Yangtian  and
      Guha, Arjun  and
      Feldman, Molly Q  and
      Anderson, Carolyn Jane",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2024",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2024.findings-acl.501/",
    doi = "10.18653/v1/2024.findings-acl.501",
    pages = "8452--8474",
    abstract = "Code LLMs have the potential to make it easier for non-experts to understand and write code. However, current CodeLLM benchmarks rely on a single expert-written prompt per problem, making it hard to generalize their success to non-expert users. In this paper, we present a new natural-language-to-code benchmark of prompts written by a key population of non-experts: beginning programmers. StudentEval contains 1,749 prompts written by 80 students who have only completed one introductory Python course. StudentEval contains numerous non-expert prompts describing the same problem, enabling exploration of key factors in prompt success. We use StudentEval to evaluate 12 Code LLMs and find that StudentEval is a better discriminator of model performance than existing benchmarks. Our analysis of student prompting strategies reveals that nondeterministic LLM sampling can mislead students about the quality of their descriptions, a finding with key implications for Code LLMs in education."
}