@inproceedings{zhang-etal-2025-direct,
    title = "Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward",
    author = "Zhang, Ruohong  and
      Gui, Liangke  and
      Sun, Zhiqing  and
      Feng, Yihao  and
      Xu, Keyang  and
      Zhang, Yuanhan  and
      Fu, Di  and
      Li, Chunyuan  and
      Hauptmann, Alexander G  and
      Bisk, Yonatan  and
      Yang, Yiming",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/Ingest-2025-COMPUTEL/2025.naacl-long.30/",
    pages = "694--717",
    ISBN = "979-8-89176-189-6",
    abstract = "Preference modeling techniques, such as direct preference optimization (DPO), has shown effective in enhancing the generalization abilities of large language model (LLM). However, in tasks involving video instruction-following, providing informative feedback, especially for open-ended conversations, remains a significant challenge. While previous studies have explored using large multimodal models (LMMs) as reward models for guiding preference modeling, their ability to accurately assess the quality of generated responses and their alignment with video content has not been conclusively demonstrated. This paper introduces a novel framework that utilizes detailed video captions as a proxy of video content, enabling language models to incorporate this information as supporting evidence for scoring video Question Answering (QA) predictions. Our approach demonstrates robust alignment with OpenAI GPT-4V model`s reward mechanism, which directly takes video frames as input. Furthermore, we show that applying our reward mechanism to DPO algorithm significantly improves model performance on open-ended video QA tasks."
}