Abstract
Modular neural networks without additional training have recently been shown to surpass end-to-end neural networks on challenging vision–language tasks. The latest such methods simultaneously introduce LLM-based code generation to build programs and a number of skill-specific, task-oriented modules to execute them. In this paper, we focus on ViperGPT and ask where its additional performance comes from and how much is due to the (state-of-art, end-to-end) BLIP-2 model it subsumes vs. additional symbolic components. To do so, we conduct a controlled study (comparing end-to-end, modular, and prompting-based methods across several VQA benchmarks). We find that ViperGPT’s reported gains over BLIP-2 can be attributed to its selection of task-specific modules, and when we run ViperGPT using a more task-agnostic selection of modules, these gains go away. ViperGPT retains much of its performance if we make prominent alterations to its selection of modules: e.g. removing or retaining only BLIP-2. We also compare ViperGPT against a prompting-based decomposition strategy and find that, on some benchmarks, modular approaches significantly benefit by representing subtasks with natural language, instead of code. Our code is fully available at https://github.com/brown-palm/visual-question-decomposition.- Anthology ID:
- 2023.emnlp-main.157
- Volume:
- Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2023
- Address:
- Singapore
- Editors:
- Houda Bouamor, Juan Pino, Kalika Bali
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2590–2603
- Language:
- URL:
- https://aclanthology.org/2023.emnlp-main.157
- DOI:
- 10.18653/v1/2023.emnlp-main.157
- Cite (ACL):
- Apoorv Khandelwal, Ellie Pavlick, and Chen Sun. 2023. Analyzing Modular Approaches for Visual Question Decomposition. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2590–2603, Singapore. Association for Computational Linguistics.
- Cite (Informal):
- Analyzing Modular Approaches for Visual Question Decomposition (Khandelwal et al., EMNLP 2023)
- PDF:
- https://preview.aclanthology.org/corrections-2024-05/2023.emnlp-main.157.pdf