@inproceedings{lin-etal-2025-intellicockpitbench,
    title = "{I}ntelli{C}ockpit{B}ench: A Comprehensive Benchmark to Evaluate {VLM}s for Intelligent Cockpit",
    author = "Lin, Liang  and
      Chai, Siyuan  and
      Wu, Jiahao  and
      Hu, Hongbing  and
      Gu, Xiaotao  and
      Hu, Hao  and
      Zhang, Fan  and
      Wang, Wei  and
      Zhang, Dan",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/display_plenaries/2025.findings-acl.798/",
    pages = "15453--15475",
    ISBN = "979-8-89176-256-5",
    abstract = "The integration of sophisticated Vision-Language Models (VLMs) in vehicular systems is revolutionizing vehicle interaction and safety, performing tasks such as Visual Question Answering (VQA). However, a critical gap persists due to the lack of a comprehensive benchmark for multimodal VQA models in vehicular scenarios. To address this, we propose IntelliCockpitBench, a benchmark that encompasses diverse automotive scenarios. It includes images from front, side, and rear cameras, various road types, weather conditions, and interior views, integrating data from both moving and stationary states. Notably, all images and queries in the benchmark are verified for high levels of authenticity, ensuring the data accurately reflects real-world conditions. A sophisticated scoring methodology combining human and model-generated assessments enhances reliability and consistency. Our contributions include a diverse and authentic dataset for automotive VQA and a robust evaluation metric aligning human and machine assessments. All code and data can be found at \url{https://github.com/Lane315/IntelliCockpitBench}."
}