Abstract
Visual Question Generation (VQG), the task of generating a question based on image contents, is an increasingly important area that combines natural language processing and computer vision. Although there are some recent works that have attempted to generate questions from images in the open domain, the task of VQG in the medical domain has not been explored so far. In this paper, we introduce an approach to generation of visual questions about radiology images called VQGR, i.e. an algorithm that is able to ask a question when shown an image. VQGR first generates new training data from the existing examples, based on contextual word embeddings and image augmentation techniques. It then uses the variational auto-encoders model to encode images into a latent space and decode natural language questions. Experimental automatic evaluations performed on the VQA-RAD dataset of clinical visual questions show that VQGR achieves good performances compared with the baseline system. The source code is available at https://github.com/sarrouti/vqgr.- Anthology ID:
- 2020.alvr-1.3
- Volume:
- Proceedings of the First Workshop on Advances in Language and Vision Research
- Month:
- July
- Year:
- 2020
- Address:
- Online
- Editors:
- Xin Wang, Jesse Thomason, Ronghang Hu, Xinlei Chen, Peter Anderson, Qi Wu, Asli Celikyilmaz, Jason Baldridge, William Yang Wang
- Venue:
- ALVR
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 12–18
- Language:
- URL:
- https://aclanthology.org/2020.alvr-1.3
- DOI:
- 10.18653/v1/2020.alvr-1.3
- Cite (ACL):
- Mourad Sarrouti, Asma Ben Abacha, and Dina Demner-Fushman. 2020. Visual Question Generation from Radiology Images. In Proceedings of the First Workshop on Advances in Language and Vision Research, pages 12–18, Online. Association for Computational Linguistics.
- Cite (Informal):
- Visual Question Generation from Radiology Images (Sarrouti et al., ALVR 2020)
- PDF:
- https://preview.aclanthology.org/emnlp-22-attachments/2020.alvr-1.3.pdf
- Code
- sarrouti/vqgr