@inproceedings{diera-etal-2023-gencodesearchnet,
    title = "{G}en{C}ode{S}earch{N}et: A Benchmark Test Suite for Evaluating Generalization in Programming Language Understanding",
    author = "Diera, Andor  and
      Dahou, Abdelhalim  and
      Galke, Lukas  and
      Karl, Fabian  and
      Sihler, Florian  and
      Scherp, Ansgar",
    editor = "Hupkes, Dieuwke  and
      Dankers, Verna  and
      Batsuren, Khuyagbaatar  and
      Sinha, Koustuv  and
      Kazemnejad, Amirhossein  and
      Christodoulopoulos, Christos  and
      Cotterell, Ryan  and
      Bruni, Elia",
    booktitle = "Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.genbench-1.2",
    doi = "10.18653/v1/2023.genbench-1.2",
    pages = "12--24",
    abstract = "Language models can serve as a valuable tool for software developers to increase productivity. Large generative models can be used for code generation and code completion, while smaller encoder-only models are capable of performing code search tasks using natural language queries. These capabilities are heavily influenced by the quality and diversity of the available training data. Source code datasets used for training usually focus on the most popular languages and testing is mostly conducted on the same distributions, often overlooking low-resource programming languages. Motivated by the NLP generalization taxonomy proposed by Hupkes et.,al., we propose a new benchmark dataset called GenCodeSearchNet (GeCS) which builds upon existing natural language code search datasets to systemically evaluate the programming language understanding generalization capabilities of language models. As part of the full dataset, we introduce a new, manually curated subset StatCodeSearch that focuses on R, a popular but so far underrepresented programming language that is often used by researchers outside the field of computer science. For evaluation and comparison, we collect several baseline results using fine-tuned BERT-style models and GPT-style large language models in a zero-shot setting.",
}