Evaluating Retrieval for Multi-domain Scientific Publications

Nancy Ide, Keith Suderman, Jingxuan Tu, Marc Verhagen, Shanan Peters, Ian Ross, John Lawson, Andrew Borg, James Pustejovsky


Abstract
This paper provides an overview of the xDD/LAPPS Grid framework and provides results of evaluating the AskMe retrievalengine using the BEIR benchmark datasets. Our primary goal is to determine a solid baseline of performance to guide furtherdevelopment of our retrieval capabilities. Beyond this, we aim to dig deeper to determine when and why certain approachesperform well (or badly) on both in-domain and out-of-domain data, an issue that has to date received relatively little attention.
Anthology ID:
2022.lrec-1.487
Volume:
Proceedings of the Thirteenth Language Resources and Evaluation Conference
Month:
June
Year:
2022
Address:
Marseille, France
Editors:
Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Jan Odijk, Stelios Piperidis
Venue:
LREC
SIG:
Publisher:
European Language Resources Association
Note:
Pages:
4569–4576
Language:
URL:
https://aclanthology.org/2022.lrec-1.487
DOI:
Bibkey:
Cite (ACL):
Nancy Ide, Keith Suderman, Jingxuan Tu, Marc Verhagen, Shanan Peters, Ian Ross, John Lawson, Andrew Borg, and James Pustejovsky. 2022. Evaluating Retrieval for Multi-domain Scientific Publications. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4569–4576, Marseille, France. European Language Resources Association.
Cite (Informal):
Evaluating Retrieval for Multi-domain Scientific Publications (Ide et al., LREC 2022)
Copy Citation:
PDF:
https://preview.aclanthology.org/nschneid-patch-1/2022.lrec-1.487.pdf
Data
BEIRBioASQTREC-COVID