Abstract
Reading comprehension style question-answering (QA) based on patient-specific documents represents a growing area in clinical NLP with plentiful applications. Bidirectional Encoder Representations from Transformers (BERT) and its derivatives lead the state-of-the-art accuracy on the task, but most evaluation has treated the data as a pre-mixture without systematically looking into the potential effect of imperfect train/test questions. The current study seeks to address this gap by experimenting with full versus partial train/test data consisting of paraphrastic questions. Our key findings include 1) training with all pooled question variants yielded best accuracy, 2) the accuracy varied widely, from 0.74 to 0.80, when trained with each single question variant, and 3) questions of similar lexical/syntactic structure tended to induce identical answers. The results suggest that how you ask questions matters in BERT-based QA, especially at the training stage.- Anthology ID:
- 2020.clinicalnlp-1.13
- Volume:
- Proceedings of the 3rd Clinical Natural Language Processing Workshop
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Editors:
- Anna Rumshisky, Kirk Roberts, Steven Bethard, Tristan Naumann
- Venue:
- ClinicalNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 111–116
- Language:
- URL:
- https://aclanthology.org/2020.clinicalnlp-1.13
- DOI:
- 10.18653/v1/2020.clinicalnlp-1.13
- Cite (ACL):
- Sungrim (Riea) Moon and Jungwei Fan. 2020. How You Ask Matters: The Effect of Paraphrastic Questions to BERT Performance on a Clinical SQuAD Dataset. In Proceedings of the 3rd Clinical Natural Language Processing Workshop, pages 111–116, Online. Association for Computational Linguistics.
- Cite (Informal):
- How You Ask Matters: The Effect of Paraphrastic Questions to BERT Performance on a Clinical SQuAD Dataset (Moon & Fan, ClinicalNLP 2020)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2020.clinicalnlp-1.13.pdf
- Data
- SQuAD, emrQA