RECOR: Reasoning-focused Multi-turn Conversational Retrieval Benchmark

Mohammed Ali, Abdelrahman Abdallah, Amit Agarwal, Hitesh Laxmichand Patel, Adam Jatowt


Abstract
Existing benchmarks treat multi-turn conversation and reasoning-intensive retrieval separately, yet real-world information seeking requires both. To bridge this gap, we present a benchmark for reasoning-based conversational information retrieval comprising 707 conversations (2,971 turns) across eleven domains. To ensure quality, our Decomposition-and-Verification framework transforms complex queries into fact-grounded multi-turn dialogues through multi-level validation, where atomic facts are verified against sources and explicit retrieval reasoning is generated for each turn. Comprehensive evaluation reveals that combining conversation history with reasoning doubles retrieval performance (Baseline .236 History+Reasoning .479 nDCG@10), while reasoning-specialized models substantially outperform dense encoders. Despite these gains, further analysis highlights that implicit reasoning remains challenging, particularly when logical connections are not explicitly stated in the text. [<https://github.com/RECOR-Benchmark/RECOR>]
Anthology ID:
2026.findings-acl.129
Volume:
Findings of the Association for Computational Linguistics: ACL 2026
Month:
July
Year:
2026
Address:
San Diego, California, United States
Editors:
Maria Liakata, Viviane P. Moreira, Jiajun Zhang, David Jurgens
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2688–2723
Language:
URL:
https://preview.aclanthology.org/ingest-acl/2026.findings-acl.129/
DOI:
Bibkey:
Cite (ACL):
Mohammed Ali, Abdelrahman Abdallah, Amit Agarwal, Hitesh Laxmichand Patel, and Adam Jatowt. 2026. RECOR: Reasoning-focused Multi-turn Conversational Retrieval Benchmark. In Findings of the Association for Computational Linguistics: ACL 2026, pages 2688–2723, San Diego, California, United States. Association for Computational Linguistics.
Cite (Informal):
RECOR: Reasoning-focused Multi-turn Conversational Retrieval Benchmark (Ali et al., Findings 2026)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingest-acl/2026.findings-acl.129.pdf
Checklist:
 2026.findings-acl.129.checklist.pdf