Abstract
Given a word in context, the task of VisualWord Sense Disambiguation consists of select-ing the correct image among a set of candidates. To select the correct image, we propose a so-lution blending text augmentation and multi-modal models. Text augmentation leverages thefine-grained semantic annotation from Word-Net to get a better representation of the tex-tual component. We then compare this sense-augmented text to the set of image using pre-trained multimodal models CLIP and ViLT. Oursystem has been ranked 16th for the Englishlanguage, achieving 68.5 points for hit rate and79.2 for mean reciprocal rank.- Anthology ID:
- 2023.semeval-1.219
- Volume:
- Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023)
- Month:
- July
- Year:
- 2023
- Address:
- Toronto, Canada
- Editors:
- Atul Kr. Ojha, A. Seza Doğruöz, Giovanni Da San Martino, Harish Tayyar Madabushi, Ritesh Kumar, Elisa Sartori
- Venue:
- SemEval
- SIG:
- SIGLEX
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1592–1597
- Language:
- URL:
- https://aclanthology.org/2023.semeval-1.219
- DOI:
- 10.18653/v1/2023.semeval-1.219
- Cite (ACL):
- Shibingfeng Zhang, Shantanu Nath, and Davide Mazzaccara. 2023. GPL at SemEval-2023 Task 1: WordNet and CLIP to Disambiguate Images. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 1592–1597, Toronto, Canada. Association for Computational Linguistics.
- Cite (Informal):
- GPL at SemEval-2023 Task 1: WordNet and CLIP to Disambiguate Images (Zhang et al., SemEval 2023)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-1/2023.semeval-1.219.pdf