Feature Difference Makes Sense: A medical image captioning model exploiting feature difference and tag information
Hyeryun Park, Kyungmo Kim, Jooyoung Yoon, Seongkeun Park, Jinwook Choi
Abstract
Medical image captioning can reduce the workload of physicians and save time and expense by automatically generating reports. However, current datasets are small and limited, creating additional challenges for researchers. In this study, we propose a feature difference and tag information combined long short-term memory (LSTM) model for chest x-ray report generation. A feature vector extracted from the image conveys visual information, but its ability to describe the image is limited. Other image captioning studies exhibited improved performance by exploiting feature differences, so the proposed model also utilizes them. First, we propose a difference and tag (DiTag) model containing the difference between the patient and normal images. Then, we propose a multi-difference and tag (mDiTag) model that also contains information about low-level differences, such as contrast, texture, and localized area. Evaluation of the proposed models demonstrates that the mDiTag model provides more information to generate captions and outperforms all other models.- Anthology ID:
- 2020.acl-srw.14
- Volume:
- Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Editors:
- Shruti Rijhwani, Jiangming Liu, Yizhong Wang, Rotem Dror
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 95–102
- Language:
- URL:
- https://aclanthology.org/2020.acl-srw.14
- DOI:
- 10.18653/v1/2020.acl-srw.14
- Cite (ACL):
- Hyeryun Park, Kyungmo Kim, Jooyoung Yoon, Seongkeun Park, and Jinwook Choi. 2020. Feature Difference Makes Sense: A medical image captioning model exploiting feature difference and tag information. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pages 95–102, Online. Association for Computational Linguistics.
- Cite (Informal):
- Feature Difference Makes Sense: A medical image captioning model exploiting feature difference and tag information (Park et al., ACL 2020)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2020.acl-srw.14.pdf