Optimal Neural Program Synthesis from Multimodal Specifications

Xi Ye, Qiaochu Chen, Isil Dillig, Greg Durrett


Abstract
Multimodal program synthesis, which leverages different types of user input to synthesize a desired program, is an attractive way to scale program synthesis to challenging settings; however, it requires integrating noisy signals from the user, like natural language, with hard constraints on the program’s behavior. This paper proposes an optimal neural synthesis approach where the goal is to find a program that satisfies user-provided constraints while also maximizing the program’s score with respect to a neural model. Specifically, we focus on multimodal synthesis tasks in which the user intent is expressed using a combination of natural language (NL) and input-output examples. At the core of our method is a top-down recurrent neural model that places distributions over abstract syntax trees conditioned on the NL input. This model not only allows for efficient search over the space of syntactically valid programs, but it allows us to leverage automated program analysis techniques for pruning the search space based on infeasibility of partial programs with respect to the user’s constraints. The experimental results on a multimodal synthesis dataset (StructuredRegex) show that our method substantially outperforms prior state-of-the-art techniques in terms of accuracy and efficiency, and finds model-optimal programs more frequently.
Anthology ID:
2021.findings-emnlp.146
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2021
Month:
November
Year:
2021
Address:
Punta Cana, Dominican Republic
Venue:
Findings
SIG:
SIGDAT
Publisher:
Association for Computational Linguistics
Note:
Pages:
1691–1704
Language:
URL:
https://aclanthology.org/2021.findings-emnlp.146
DOI:
10.18653/v1/2021.findings-emnlp.146
Bibkey:
Cite (ACL):
Xi Ye, Qiaochu Chen, Isil Dillig, and Greg Durrett. 2021. Optimal Neural Program Synthesis from Multimodal Specifications. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1691–1704, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
Optimal Neural Program Synthesis from Multimodal Specifications (Ye et al., Findings 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/auto-file-uploads/2021.findings-emnlp.146.pdf
Video:
 https://preview.aclanthology.org/auto-file-uploads/2021.findings-emnlp.146.mp4