Image2tweet: Datasets in Hindi and English for Generating Tweets from Images
Rishabh Jha, Varshith Kaki, Varuna Kolla, Shubham Bhagat, Parth Patwa, Amitava Das, Santanu Pal
Abstract
Image Captioning as a task that has seen major updates over time. In recent methods, visual-linguistic grounding of the image-text pair is leveraged. This includes either generating the textual description of the objects and entities present within the image in constrained manner, or generating detailed description of these entities as a paragraph. But there is still a long way to go towards being able to generate text that is not only semantically richer, but also contains real world knowledge in it. This is the motivation behind exploring image2tweet generation through the lens of existing image-captioning approaches. At the same time, there is little research in image captioning in Indian languages like Hindi. In this paper, we release Hindi and English datasets for the task of tweet generation given an image. The aim is to generate a specialized text like a tweet, that is not a direct result of visual-linguistic grounding that is usually leveraged in similar tasks, but conveys a message that factors-in not only the visual content of the image, but also additional real world contextual information associated with the event described within the image as closely as possible. Further, We provide baseline DL models on our data and invite researchers to build more sophisticated systems for the problem.- Anthology ID:
- 2021.icon-main.84
- Volume:
- Proceedings of the 18th International Conference on Natural Language Processing (ICON)
- Month:
- December
- Year:
- 2021
- Address:
- National Institute of Technology Silchar, Silchar, India
- Editors:
- Sivaji Bandyopadhyay, Sobha Lalitha Devi, Pushpak Bhattacharyya
- Venue:
- ICON
- SIG:
- Publisher:
- NLP Association of India (NLPAI)
- Note:
- Pages:
- 670–676
- Language:
- URL:
- https://aclanthology.org/2021.icon-main.84
- DOI:
- Cite (ACL):
- Rishabh Jha, Varshith Kaki, Varuna Kolla, Shubham Bhagat, Parth Patwa, Amitava Das, and Santanu Pal. 2021. Image2tweet: Datasets in Hindi and English for Generating Tweets from Images. In Proceedings of the 18th International Conference on Natural Language Processing (ICON), pages 670–676, National Institute of Technology Silchar, Silchar, India. NLP Association of India (NLPAI).
- Cite (Informal):
- Image2tweet: Datasets in Hindi and English for Generating Tweets from Images (Jha et al., ICON 2021)
- PDF:
- https://preview.aclanthology.org/landing_page/2021.icon-main.84.pdf
- Code
- git-rishabh-jha/image2tweet
- Data
- Flickr30k, MS COCO