@inproceedings{purohit-etal-2024-explora,
    title = "{EXPLORA}: Efficient Exemplar Subset Selection for Complex Reasoning",
    author = "Purohit, Kiran  and
      V, Venktesh  and
      Devalla, Raghuram  and
      Yerragorla, Krishna Mohan  and
      Bhattacharya, Sourangshu  and
      Anand, Avishek",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2024.emnlp-main.307/",
    doi = "10.18653/v1/2024.emnlp-main.307",
    pages = "5367--5388",
    abstract = "Answering reasoning-based complex questions over text and hybrid sources, including tables, is a challenging task. Recent advances in large language models (LLMs) have enabled in-context learning (ICL), allowing LLMs to acquire proficiency in a specific task using only a few demonstration samples (exemplars). A critical challenge in ICL is the selection of optimal exemplars, which can be either task-specific (static) or test-example-specific (dynamic). Static exemplars provide faster inference times and increased robustness across a distribution of test examples. In this paper, we propose an algorithm for static exemplar subset selection for complex reasoning tasks. We introduce EXPLORA, a novel exploration method designed to estimate the parameters of the scoring function, which evaluates exemplar subsets without incorporating confidence information. EXPLORA significantly reduces the number of LLM calls to {\textasciitilde}11{\%} of those required by state-of-the-art methods and achieves a substantial performance improvement of 12.24{\%}. We open-source our code and data (https://github.com/kiranpurohit/EXPLORA)."
}