Abstract
Visual storytelling (VST) is the task of generating a story paragraph that describes a given image sequence. Most existing storytelling approaches have evaluated their models using traditional natural language generation metrics like BLEU or CIDEr. However, such metrics based on n-gram matching tend to have poor correlation with human evaluation scores and do not explicitly consider other criteria necessary for storytelling such as sentence structure or topic coherence. Moreover, a single score is not enough to assess a story as it does not inform us about what specific errors were made by the model. In this paper, we propose 3 evaluation metrics sets that analyses which aspects we would look for in a good story: 1) visual grounding, 2) coherence, and 3) non-redundancy. We measure the reliability of our metric sets by analysing its correlation with human judgement scores on a sample of machine stories obtained from 4 state-of-the-arts models trained on the Visual Storytelling Dataset (VIST). Our metric sets outperforms other metrics on human correlation, and could be served as a learning based evaluation metric set that is complementary to existing rule-based metrics.- Anthology ID:
- 2022.findings-naacl.206
- Volume:
- Findings of the Association for Computational Linguistics: NAACL 2022
- Month:
- July
- Year:
- 2022
- Address:
- Seattle, United States
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2691–2702
- Language:
- URL:
- https://aclanthology.org/2022.findings-naacl.206
- DOI:
- 10.18653/v1/2022.findings-naacl.206
- Cite (ACL):
- Eileen Wang, Caren Han, and Josiah Poon. 2022. RoViST: Learning Robust Metrics for Visual Storytelling. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2691–2702, Seattle, United States. Association for Computational Linguistics.
- Cite (Informal):
- RoViST: Learning Robust Metrics for Visual Storytelling (Wang et al., Findings 2022)
- PDF:
- https://preview.aclanthology.org/ingestion-script-update/2022.findings-naacl.206.pdf
- Code
- usydnlp/rovist
- Data
- Flickr30k, ROCStories, VIST