Abstract
Multimodal Summarization (MS) has attracted research interest in the past few years due to the ease with which users perceive multimodal summaries. It is important for MS models to consider the topic a given target content belongs to. In the current paper, we propose a topic-aware MS system which performs two tasks simultaneously: differentiating the images into “on-topic” and “off-topic” categories and further utilizing the “on-topic” images to generate multimodal summaries. The hypothesis is that, the proposed topic similarity classifier will help in generating better multimodal summary by focusing on important components of images and text which are specific to a particular topic. To develop the topic similarity classifier, we have augmented the existing popular MS data set, MSMO, with similar “on-topic” and dissimilar “off-topic” images for each sample. Our experimental results establish that the focus on “on-topic” features helps in generating topic-aware multimodal summaries, which outperforms the state of the art approach by 1.7 % in ROUGE-L metric.- Anthology ID:
- 2022.findings-aacl.36
- Volume:
- Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022
- Month:
- November
- Year:
- 2022
- Address:
- Online only
- Venue:
- Findings
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 387–398
- Language:
- URL:
- https://aclanthology.org/2022.findings-aacl.36
- DOI:
- Cite (ACL):
- Sourajit Mukherjee, Anubhav Jangra, Sriparna Saha, and Adam Jatowt. 2022. Topic-aware Multimodal Summarization. In Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022, pages 387–398, Online only. Association for Computational Linguistics.
- Cite (Informal):
- Topic-aware Multimodal Summarization (Mukherjee et al., Findings 2022)
- PDF:
- https://preview.aclanthology.org/auto-file-uploads/2022.findings-aacl.36.pdf