@inproceedings{zhong-etal-2023-non,
    title = "Non-Programmers Can Label Programs Indirectly via Active Examples: A Case Study with Text-to-{SQL}",
    author = "Zhong, Ruiqi  and
      Snell, Charlie  and
      Klein, Dan  and
      Eisner, Jason",
    editor = "Bouamor, Houda  and
      Pino, Juan  and
      Bali, Kalika",
    booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Author-page-Marten-During-lu/2023.emnlp-main.312/",
    doi = "10.18653/v1/2023.emnlp-main.312",
    pages = "5126--5152",
    abstract = "Can non-programmers annotate natural language utterances with complex programs that represent their meaning? We introduce APEL, a framework in which non-programmers select among candidate programs generated by a seed semantic parser (e.g., Codex). Since they cannot understand the candidate programs, we ask them to select indirectly by examining the programs' input-ouput examples. For each utterance, APEL actively searches for a simple input on which the candidate programs tend to produce different outputs. It then asks the non-programmers only to choose the appropriate output, thus allowing us to infer which program is correct and could be used to fine-tune the parser. As a first case study, we recruited human non-programmers to use APEL to re-annotate SPIDER, a text-to-SQL dataset. Our approach achieved the same annotation accuracy as the original expert annotators (75{\%}) and exposed many subtle errors in the original annotations."
}