Abstract
Multi-modal large language models (MLLMs) are expected to support multi-turn queries of interchanging image and text modalities in production. However, the current MLLMs trained with visual-question-answering (VQA) datasets could suffer from degradation, as VQA datasets lack the diversity and complexity of the original text instruction datasets with which the underlying language model was trained. To address this degradation, we first collect a lightweight, 5k-sample VQA preference dataset where answers were annotated by Gemini for five quality metrics in a granular fashion and investigate standard Supervised Fine-tuning, rejection sampling, Direct Preference Optimization (DPO) and SteerLM algorithms. Our findings indicate that with DPO, we can surpass the instruction-following capabilities of the language model, achieving a 6.73 score on MT-Bench, compared to Vicuna’s 6.57 and LLaVA’s 5.99. This enhancement in textual instruction-following capability correlates with boosted visual instruction performance (+4.9% on MM-Vet, +6% on LLaVA-Bench), with minimal alignment tax on visual knowledge benchmarks compared to the previous RLHF approach. In conclusion, we propose a distillation-based multi-modal alignment model with fine-grained annotations on a small dataset that restores and boosts MLLM’s language capability after visual instruction tuning.- Anthology ID:
- 2024.acl-long.765
- Volume:
- Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
- Month:
- August
- Year:
- 2024
- Address:
- Bangkok, Thailand
- Editors:
- Lun-Wei Ku, Andre Martins, Vivek Srikumar
- Venue:
- ACL
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 14188–14200
- Language:
- URL:
- https://preview.aclanthology.org/icon-24-ingestion/2024.acl-long.765/
- DOI:
- 10.18653/v1/2024.acl-long.765
- Cite (ACL):
- Shengzhi Li, Rongyu Lin, and Shichao Pei. 2024. Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14188–14200, Bangkok, Thailand. Association for Computational Linguistics.
- Cite (Informal):
- Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Models (Li et al., ACL 2024)
- PDF:
- https://preview.aclanthology.org/icon-24-ingestion/2024.acl-long.765.pdf