Abstract
Is aesthetic impact different from beauty? Is visual salience a reflection of its capacity for effective communication? We present Impressions, a novel dataset through which to investigate the semiotics of images, and how specific visual features and design choices can elicit specific emotions, thoughts and beliefs. We posit that the impactfulness of an image extends beyond formal definitions of aesthetics, to its success as a communicative act, where style contributes as much to meaning formation as the subject matter. We also acknowledge that existing Image Captioning datasets are not designed to empower state-of-the-art architectures to model potential human impressions or interpretations of images. To fill this need, we design an annotation task heavily inspired by image analysis techniques in the Visual Arts to collect 1,440 image-caption pairs and 4,320 unique annotations exploring impact, pragmatic image description, impressions and aesthetic design choices. We show that existing multimodal image captioning and conditional generation models struggle to simulate plausible human responses to images. However, this dataset significantly improves their ability to model impressions and aesthetic evaluations of images through fine-tuning and few-shot adaptation.- Anthology ID:
- 2023.emnlp-main.755
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 12273–12291
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-main.755
- DOI:
- 10.18653/v1/2023.emnlp-main.755
- Cite (ACL):
- Julia Kruk, Caleb Ziems, and Diyi Yang. 2023. Impressions: Visual Semiotics and Aesthetic Impact Understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12273–12291, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Impressions: Visual Semiotics and Aesthetic Impact Understanding (Kruk et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/improve-issue-templates/2023.emnlp-main.755.pdf