ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation
Wanrong Zhu, Xin Wang, An Yan, Miguel Eckstein, William Yang Wang
Abstract
Automatic evaluations for natural language generation (NLG) conventionally rely on token-level or embedding-level comparisons with text references. This differs from human language processing, for which visual imagination often improves comprehension. In this work, we propose ImaginE, an imagination-based automatic evaluation metric for natural language generation. With the help of StableDiffusion, a state-of-the-art text-to-image generator, we automatically generate an image as the embodied imagination for the text snippet and compute the imagination similarity using contextual embeddings. Experiments spanning several text generation tasks demonstrate that adding machine-generated images with our ImaginE displays great potential in introducing multi-modal information into NLG evaluation, and improves existing automatic metrics’ correlations with human similarity judgments in both reference-based and reference-free evaluation scenarios.- Anthology ID:
- 2023.findings-eacl.6
- Volume:
- Findings of the Association for Computational Linguistics: EACL 2023
- Month:
- May
- Year:
- 2023
- Address:
- Dubrovnik, Croatia
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 93–105
- Language:
- URL:
- https://aclanthology.org/2023.findings-eacl.6
- DOI:
- Cite (ACL):
- Wanrong Zhu, Xin Wang, An Yan, Miguel Eckstein, and William Yang Wang. 2023. ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation. In Findings of the Association for Computational Linguistics: EACL 2023, pages 93–105, Dubrovnik, Croatia. Association for Computational Linguistics.
- Cite (Informal):
- ImaginE: An Imagination-Based Automatic Evaluation Metric for Natural Language Generation (Zhu et al., Findings 2023)
- PDF:
- https://preview.aclanthology.org/nodalida-main-page/2023.findings-eacl.6.pdf