@inproceedings{ma-etal-2025-latte,
    title = "{LATTE}: Learning to Think with Vision Specialists",
    author = "Ma, Zixian  and
      Zhang, Jianguo  and
      Liu, Zhiwei  and
      Zhang, Jieyu  and
      Tan, Juntao  and
      Shu, Manli  and
      Niebles, Juan Carlos  and
      Heinecke, Shelby  and
      Wang, Huan  and
      Xiong, Caiming  and
      Krishna, Ranjay  and
      Savarese, Silvio",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.emnlp-main.564/",
    doi = "10.18653/v1/2025.emnlp-main.564",
    pages = "11203--11240",
    ISBN = "979-8-89176-332-6",
    abstract = "While open-source vision-language models perform well on simple question-answering, they still struggle with complex questions that require both perceptual and reasoning capabilities. We propose LATTE, a family of vision-language models that have LeArned to Think wiTh vision spEcialists. By offloading perception to state-of-the-art vision models, our approach enables vision-language models to focus solely on reasoning over high-quality perceptual information. To train LATTE, we synthesize and filter a large dataset of 293K multi-modal reasoning traces over perceptual outputs of vision specialists. LATTE trained on this data achieves significant 4-5{\%} gains over baselines across 6 benchmarks covering both perception and reasoning abilities. Ablation studies reveal that the effectiveness of multi-modal reasoning traces depends on the data sources, formats, and quality of thoughts."
}