♫ MuSiQue: Multihop Questions via Single-hop Question Composition
Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal
Abstract
Multihop reasoning remains an elusive goal as existing multihop benchmarks are known to be largely solvable via shortcuts. Can we create a question answering (QA) dataset that, by construction, requires proper multihop reasoning? To this end, we introduce a bottom–up approach that systematically selects composable pairs of single-hop questions that are connected, that is, where one reasoning step critically relies on information from another. This bottom–up methodology lets us explore a vast space of questions and add stringent filters as well as other mechanisms targeting connected reasoning. It provides fine-grained control over the construction process and the properties of the resulting k-hop questions. We use this methodology to create MuSiQue-Ans, a new multihop QA dataset with 25K 2–4 hop questions. Relative to existing datasets, MuSiQue-Ans is more difficult overall (3× increase in human–machine gap), and harder to cheat via disconnected reasoning (e.g., a single-hop model has a 30-point drop in F1). We further add unanswerable contrast questions to produce a more stringent dataset, MuSiQue-Full. We hope our datasets will help the NLP community develop models that perform genuine multihop reasoning.1- Anthology ID:
- 2022.tacl-1.31
- Volume:
- Transactions of the Association for Computational Linguistics, Volume 10
- Month:
- Year:
- 2022
- Address:
- Cambridge, MA
- Venue:
- TACL
- SIG:
- Publisher:
- MIT Press
- Note:
- Pages:
- 539–554
- Language:
- URL:
- https://aclanthology.org/2022.tacl-1.31
- DOI:
- 10.1162/tacl_a_00475
- Cite (ACL):
- Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. ♫ MuSiQue: Multihop Questions via Single-hop Question Composition. Transactions of the Association for Computational Linguistics, 10:539–554.
- Cite (Informal):
- ♫ MuSiQue: Multihop Questions via Single-hop Question Composition (Trivedi et al., TACL 2022)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2022.tacl-1.31.pdf