Drilling Down into the Discourse Structure with LLMs for Long Document Question Answering
Inderjeet Nair, Shwetha Somasundaram, Apoorv Saxena, Koustava Goswami
Abstract
We address the task of evidence retrieval for long document question answering, which involves locating relevant paragraphs within a document to answer a question. We aim to assess the applicability of large language models (LLMs) in the task of zero-shot long document evidence retrieval, owing to their unprecedented performance across various NLP tasks. However, currently the LLMs can consume limited context lengths as input, thus providing document chunks as inputs might overlook the global context while missing out on capturing the inter-segment dependencies. Moreover, directly feeding the large input sets can incur significant computational costs, particularly when processing the entire document (and potentially incurring monetary expenses with enterprise APIs like OpenAI’s GPT variants). To address these challenges, we propose a suite of techniques that exploit the discourse structure commonly found in documents. By utilizing this structure, we create a condensed representation of the document, enabling a more comprehensive understanding and analysis of relationships between different parts. We retain 99.6% of the best zero-shot approach’s performance, while processing only 26% of the total tokens used by the best approach in the information seeking evidence retrieval setup. We also show how our approach can be combined with *self-ask* reasoning agent to achieve best zero-shot performance in complex multi-hop question answering, just ≈ 4% short of zero-shot performance using gold evidence.- Anthology ID:
- 2023.findings-emnlp.972
- Volume:
- Findings of the Association for Computational Linguistics: EMNLP 2023
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 14593–14606
- Language:
- URL:
- https://aclanthology.org/2023.findings-emnlp.972
- DOI:
- 10.18653/v1/2023.findings-emnlp.972
- Cite (ACL):
- Inderjeet Nair, Shwetha Somasundaram, Apoorv Saxena, and Koustava Goswami. 2023. Drilling Down into the Discourse Structure with LLMs for Long Document Question Answering. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14593–14606, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Drilling Down into the Discourse Structure with LLMs for Long Document Question Answering (Nair et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/add_acl24_videos/2023.findings-emnlp.972.pdf