A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding
Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan Plummer, Kate Saenko, Jianmo Ni, Mandy Guo
Abstract
Webpages have been a rich, scalable resource for vision-language and language only tasks. Yet only pieces of webpages are kept in existing datasets: image-caption pairs, long text articles, or raw HTML, never all in one place. Webpage tasks have resultingly received little attention and structured image-text data left underused. To study multimodal webpage understanding, we introduce the Wikipedia Webpage suite (WikiWeb2M) containing 2M pages with all of the associated image, text, and structure data. We verify its utility on three generative tasks: page description generation, section summarization, and contextual image captioning. We design a novel attention mechanism Prefix Global, which selects the most relevant image and text content as global tokens to attend to the rest of the webpage for context. By using page structure to separate such tokens, it performs better than full attention with lower computational complexity. Extensive experiments show that the new data in WikiWeb2M improves task performance compared to prior work.- Anthology ID:
- 2023.emnlp-main.119
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 1917–1947
- Language:
- URL:
- https://preview.aclanthology.org/icon-24-ingestion/2023.emnlp-main.119/
- DOI:
- 10.18653/v1/2023.emnlp-main.119
- Cite (ACL):
- Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan Plummer, Kate Saenko, Jianmo Ni, and Mandy Guo. 2023. A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 1917–1947, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding (Burns et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/icon-24-ingestion/2023.emnlp-main.119.pdf