Evaluating Retrieval for Multi-domain Scientific Publications
Nancy Ide, Keith Suderman, Jingxuan Tu, Marc Verhagen, Shanan Peters, Ian Ross, John Lawson, Andrew Borg, James Pustejovsky
Abstract
This paper provides an overview of the xDD/LAPPS Grid framework and provides results of evaluating the AskMe retrievalengine using the BEIR benchmark datasets. Our primary goal is to determine a solid baseline of performance to guide furtherdevelopment of our retrieval capabilities. Beyond this, we aim to dig deeper to determine when and why certain approachesperform well (or badly) on both in-domain and out-of-domain data, an issue that has to date received relatively little attention.- Anthology ID:
- 2022.lrec-1.487
- Volume:
- Proceedings of the Thirteenth Language Resources and Evaluation Conference
- Month:
- June
- Year:
- 2022
- Address:
- Marseille, France
- Editors:
- Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, Stelios Piperidis
- Venue:
- LREC
- SIG:
- Publisher:
- European Language Resources Association
- Note:
- Pages:
- 4569–4576
- Language:
- URL:
- https://aclanthology.org/2022.lrec-1.487
- DOI:
- Cite (ACL):
- Nancy Ide, Keith Suderman, Jingxuan Tu, Marc Verhagen, Shanan Peters, Ian Ross, John Lawson, Andrew Borg, and James Pustejovsky. 2022. Evaluating Retrieval for Multi-domain Scientific Publications. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4569–4576, Marseille, France. European Language Resources Association.
- Cite (Informal):
- Evaluating Retrieval for Multi-domain Scientific Publications (Ide et al., LREC 2022)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/2022.lrec-1.487.pdf
- Data
- BEIR, BioASQ, TREC-COVID