How do people talk about images? A study on open-domain conversations with images.
Yi-Pei Chen, Nobuyuki Shimizu, Takashi Miyazaki, Hideki Nakayama
Abstract
This paper explores how humans conduct conversations with images by investigating an open-domain image conversation dataset, ImageChat. We examined the conversations with images from the perspectives of image relevancy and image information. We found that utterances/conversations are not always related to the given image, and conversation topics diverge within three turns about half of the time. Besides image objects, more comprehensive non-object image information is also indispensable. After inspecting the causes, we suggested that understanding the overall scenario of image and connecting objects based on their high-level attributes might be very helpful to generate more engaging open-domain conversations when an image is presented. We proposed enriching the image information with image caption and object tags based on our analysis. With our proposed image+ features, we improved automatic metrics including BLEU and Bert Score, and increased the diversity and image-relevancy of generated responses to the strong baseline. The result verifies that our analysis provides valuable insights and could facilitate future research on open-domain conversations with images.- Anthology ID:
- 2022.naacl-srw.20
- Volume:
- Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop
- Month:
- July
- Year:
- 2022
- Address:
- Hybrid: Seattle, Washington + Online
- Editors:
- Daphne Ippolito, Liunian Harold Li, Maria Leonor Pacheco, Danqi Chen, Nianwen Xue
- Venue:
- NAACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 156–162
- Language:
- URL:
- https://aclanthology.org/2022.naacl-srw.20
- DOI:
- 10.18653/v1/2022.naacl-srw.20
- Cite (ACL):
- Yi-Pei Chen, Nobuyuki Shimizu, Takashi Miyazaki, and Hideki Nakayama. 2022. How do people talk about images? A study on open-domain conversations with images.. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Student Research Workshop, pages 156–162, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics.
- Cite (Informal):
- How do people talk about images? A study on open-domain conversations with images. (Chen et al., NAACL 2022)
- PDF:
- https://preview.aclanthology.org/ingest-acl-2023-videos/2022.naacl-srw.20.pdf
- Data
- Image-Chat