Not a cute stroke: Analysis of Rule- and Neural Network-based Information Extraction Systems for Brain Radiology Reports
Andreas Grivas, Beatrice Alex, Claire Grover, Richard Tobin, William Whiteley
Abstract
We present an in-depth comparison of three clinical information extraction (IE) systems designed to perform entity recognition and negation detection on brain imaging reports: EdIE-R, a bespoke rule-based system, and two neural network models, EdIE-BiLSTM and EdIE-BERT, both multi-task learning models with a BiLSTM and BERT encoder respectively. We compare our models both on an in-sample and an out-of-sample dataset containing mentions of stroke findings and draw on our error analysis to suggest improvements for effective annotation when building clinical NLP models for a new domain. Our analysis finds that our rule-based system outperforms the neural models on both datasets and seems to generalise to the out-of-sample dataset. On the other hand, the neural models do not generalise negation to the out-of-sample dataset, despite metrics on the in-sample dataset suggesting otherwise.- Anthology ID:
- 2020.louhi-1.4
- Volume:
- Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis
- Month:
- November
- Year:
- 2020
- Address:
- Online
- Editors:
- Eben Holderness, Antonio Jimeno Yepes, Alberto Lavelli, Anne-Lyse Minard, James Pustejovsky, Fabio Rinaldi
- Venue:
- Louhi
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 24–37
- Language:
- URL:
- https://aclanthology.org/2020.louhi-1.4
- DOI:
- 10.18653/v1/2020.louhi-1.4
- Cite (ACL):
- Andreas Grivas, Beatrice Alex, Claire Grover, Richard Tobin, and William Whiteley. 2020. Not a cute stroke: Analysis of Rule- and Neural Network-based Information Extraction Systems for Brain Radiology Reports. In Proceedings of the 11th International Workshop on Health Text Mining and Information Analysis, pages 24–37, Online. Association for Computational Linguistics.
- Cite (Informal):
- Not a cute stroke: Analysis of Rule- and Neural Network-based Information Extraction Systems for Brain Radiology Reports (Grivas et al., Louhi 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-5/2020.louhi-1.4.pdf
- Code
- edinburgh-ltg/edieviz
- Data
- BLUE