Abstract
Recent studies show that neural natural processing models for medical code prediction suffer from a label imbalance issue. This study aims to investigate further imbalance in a medical code prediction dataset in terms of demographic variables and analyse performance differences in demographic groups. We use sample-based metrics to correctly evaluate the performance in terms of the data subject. Also, a simple label distance metric is proposed to quantify the difference in the label distribution between a group and the entire data. Our analysis results reveal that the model performs differently towards different demographic groups: significant differences between age groups and between insurance types are observed. Interestingly, we found a weak positive correlation between the number of training data of the group and the performance of the group. However, a strong negative correlation between the label distance of the group and the performance of the group is observed. This result suggests that the model tends to perform poorly in the group whose label distribution is different from the global label distribution of the training data set. Further analysis of the model performance is required to identify the cause of these differences and to improve the model building.- Anthology ID:
- 2022.clinicalnlp-1.10
- Volume:
- Proceedings of the 4th Clinical Natural Language Processing Workshop
- Month:
- July
- Year:
- 2022
- Address:
- Seattle, WA
- Editors:
- Tristan Naumann, Steven Bethard, Kirk Roberts, Anna Rumshisky
- Venue:
- ClinicalNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 93–102
- Language:
- URL:
- https://aclanthology.org/2022.clinicalnlp-1.10
- DOI:
- 10.18653/v1/2022.clinicalnlp-1.10
- Cite (ACL):
- Heereen Shim, Dietwig Lowet, Stijn Luca, and Bart Vanrumste. 2022. An exploratory data analysis: the performance differences of a medical code prediction system on different demographic groups. In Proceedings of the 4th Clinical Natural Language Processing Workshop, pages 93–102, Seattle, WA. Association for Computational Linguistics.
- Cite (Informal):
- An exploratory data analysis: the performance differences of a medical code prediction system on different demographic groups (Shim et al., ClinicalNLP 2022)
- PDF:
- https://preview.aclanthology.org/naacl24-info/2022.clinicalnlp-1.10.pdf
- Data
- MIMIC-III