Faithful Chart Summarization with ChaTS-Pi
Syrine Krichene, Francesco Piccinno, Fangyu Liu, Julian Eisenschlos
Abstract
Chart-to-summary generation can help explore data, communicate insights, and help the visually impaired people. Multi-modal generative models have been used to produce fluent summaries, but they can suffer from factual and perceptual errors. In this work we present CHATS-CRITIC, a reference-free chart summarization metric for scoring faithfulness. CHATS-CRITIC is composed of an image-to-text model to recover the table from a chart, and a tabular entailment model applied to score the summary sentence by sentence. We find that CHATS-CRITIC evaluates the summary quality according to human ratings better than reference-based metrics, either learned or n-gram based, and can be further used to fix candidate summaries by removing not supported sentences. We then introduce CHATS-PI, a chart-to-summary pipeline that leverages CHATS-CRITIC during inference to fix and rank sampled candidates from any chart-summarization model. We evaluate CHATS-PI and CHATS-CRITIC using human raters, establishing state-of-the-art results on two popular chart-to-summary datasets.- Anthology ID:
- 2024.acl-long.472
- Volume:
- Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Lun-Wei Ku, Andre Martins, Vivek Srikumar
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 8705–8723
- Language:
- URL:
- https://aclanthology.org/2024.acl-long.472
- DOI:
- Cite (ACL):
- Syrine Krichene, Francesco Piccinno, Fangyu Liu, and Julian Eisenschlos. 2024. Faithful Chart Summarization with ChaTS-Pi. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8705–8723, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- Faithful Chart Summarization with ChaTS-Pi (Krichene et al., ACL 2024)
- PDF:
- https://preview.aclanthology.org/nschneid-patch-4/2024.acl-long.472.pdf