@inproceedings{ferguson-etal-2022-retrieval,
    title = "Retrieval Data Augmentation Informed by Downstream Question Answering Performance",
    author = "Ferguson, James  and
      Hajishirzi, Hannaneh  and
      Dasigi, Pradeep  and
      Khot, Tushar",
    editor = "Aly, Rami  and
      Christodoulopoulos, Christos  and
      Cocarascu, Oana  and
      Guo, Zhijiang  and
      Mittal, Arpit  and
      Schlichtkrull, Michael  and
      Thorne, James  and
      Vlachos, Andreas",
    booktitle = "Proceedings of the Fifth Fact Extraction and VERification Workshop (FEVER)",
    month = may,
    year = "2022",
    address = "Dublin, Ireland",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2022.fever-1.1",
    doi = "10.18653/v1/2022.fever-1.1",
    pages = "1--5",
    abstract = "Training retrieval models to fetch contexts for Question Answering (QA) over large corpora requires labeling relevant passages in those corpora. Since obtaining exhaustive manual annotations of all relevant passages is not feasible, prior work uses text overlap heuristics to find passages that are likely to contain the answer, but this is not feasible when the task requires deeper reasoning and answers are not extractable spans (e.g.: multi-hop, discrete reasoning). We address this issue by identifying relevant passages based on whether they are useful for a trained QA model to arrive at the correct answers, and develop a search process guided by the QA model{'}s loss. Our experiments show that this approach enables identifying relevant context for unseen data greater than 90{\%} of the time on the IIRC dataset and generalizes better to the end QA task than those trained on just the gold retrieval data on IIRC and QASC datasets.",
}