TextGraphs 2021 Shared Task on Multi-Hop Inference for Explanation Regeneration
Mokanarangan Thayaparan, Marco Valentino, Peter Jansen, Dmitry Ustalov
Abstract
The Shared Task on Multi-Hop Inference for Explanation Regeneration asks participants to compose large multi-hop explanations to questions by assembling large chains of facts from a supporting knowledge base. While previous editions of this shared task aimed to evaluate explanatory completeness – finding a set of facts that form a complete inference chain, without gaps, to arrive from question to correct answer, this 2021 instantiation concentrates on the subtask of determining relevance in large multi-hop explanations. To this end, this edition of the shared task makes use of a large set of approximately 250k manual explanatory relevancy ratings that augment the 2020 shared task data. In this summary paper, we describe the details of the explanation regeneration task, the evaluation data, and the participating systems. Additionally, we perform a detailed analysis of participating systems, evaluating various aspects involved in the multi-hop inference process. The best performing system achieved an NDCG of 0.82 on this challenging task, substantially increasing performance over baseline methods by 32%, while also leaving significant room for future improvement.- Anthology ID:
- 2021.textgraphs-1.17
- Volume:
- Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15)
- Month:
- June
- Year:
- 2021
- Address:
- Mexico City, Mexico
- Editors:
- Alexander Panchenko, Fragkiskos D. Malliaros, Varvara Logacheva, Abhik Jana, Dmitry Ustalov, Peter Jansen
- Venue:
- TextGraphs
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 156–165
- Language:
- URL:
- https://aclanthology.org/2021.textgraphs-1.17
- DOI:
- 10.18653/v1/2021.textgraphs-1.17
- Cite (ACL):
- Mokanarangan Thayaparan, Marco Valentino, Peter Jansen, and Dmitry Ustalov. 2021. TextGraphs 2021 Shared Task on Multi-Hop Inference for Explanation Regeneration. In Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15), pages 156–165, Mexico City, Mexico. Association for Computational Linguistics.
- Cite (Informal):
- TextGraphs 2021 Shared Task on Multi-Hop Inference for Explanation Regeneration (Thayaparan et al., TextGraphs 2021)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2021.textgraphs-1.17.pdf
- Code
- cognitiveailab/tg2021task
- Data
- Worldtree