Abstract
We present Reddit Health Online Talk (RedHOT), a corpus of 22,000 richly annotated social media posts from Reddit spanning 24 health conditions. Annotations include demarcations of spans corresponding to medical claims, personal experiences, and questions. We collect additional granular annotations on identified claims. Specifically, we mark snippets that describe patient Populations, Interventions, and Outcomes (PIO elements) within these. Using this corpus, we introduce the task of retrieving trustworthy evidence relevant to a given claim made on social media. We propose a new method to automatically derive (noisy) supervision for this task which we use to train a dense retrieval model; this outperforms baseline models. Manual evaluation of retrieval results performed by medical doctors indicate that while our system performance is promising, there is considerable room for improvement. We release all annotations collected (and scripts to assemble the dataset), and all code necessary to reproduce the results in this paper at: https://sominw.com/redhot.- Anthology ID:
- 2023.findings-eacl.61
- Volume:
- Findings of the Association for Computational Linguistics: EACL 2023
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Editors:
- Andreas Vlachos, Isabelle Augenstein
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 809–827
- Language:
- URL:
- https://aclanthology.org/2023.findings-eacl.61
- DOI:
- 10.18653/v1/2023.findings-eacl.61
- Cite (ACL):
- Somin Wadhwa, Vivek Khetan, Silvio Amir, and Byron Wallace. 2023. RedHOT: A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media. In Findings of the Association for Computational Linguistics: EACL 2023, pages 809–827, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- RedHOT: A Corpus of Annotated Medical Questions, Experiences, and Claims on Social Media (Wadhwa et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/dois-2013-emnlp/2023.findings-eacl.61.pdf