COUGH: A Challenge Dataset and Models for COVID-19 FAQ Retrieval

Xinliang Frederick Zhang, Heming Sun, Xiang Yue, Simon Lin, Huan Sun


Abstract
We present a large, challenging dataset, COUGH, for COVID-19 FAQ retrieval. Similar to a standard FAQ dataset, COUGH consists of three parts: FAQ Bank, Query Bank and Relevance Set. The FAQ Bank contains ~16K FAQ items scraped from 55 credible websites (e.g., CDC and WHO). For evaluation, we introduce Query Bank and Relevance Set, where the former contains 1,236 human-paraphrased queries while the latter contains ~32 human-annotated FAQ items for each query. We analyze COUGH by testing different FAQ retrieval models built on top of BM25 and BERT, among which the best model achieves 48.8 under P@5, indicating a great challenge presented by COUGH and encouraging future research for further improvement. Our COUGH dataset is available at https://github.com/sunlab-osu/covid-faq.
Anthology ID:
2021.emnlp-main.305
Volume:
Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing
Month:
November
Year:
2021
Address:
Online and Punta Cana, Dominican Republic
Venue:
EMNLP
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3759–3769
Language:
URL:
https://aclanthology.org/2021.emnlp-main.305
DOI:
10.18653/v1/2021.emnlp-main.305
Bibkey:
Cite (ACL):
Xinliang Frederick Zhang, Heming Sun, Xiang Yue, Simon Lin, and Huan Sun. 2021. COUGH: A Challenge Dataset and Models for COVID-19 FAQ Retrieval. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3759–3769, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Cite (Informal):
COUGH: A Challenge Dataset and Models for COVID-19 FAQ Retrieval (Zhang et al., EMNLP 2021)
Copy Citation:
PDF:
https://preview.aclanthology.org/ingestion-script-update/2021.emnlp-main.305.pdf
Video:
 https://preview.aclanthology.org/ingestion-script-update/2021.emnlp-main.305.mp4
Code
 sunlab-osu/covid-faq
Data
COUGH