@inproceedings{rawte-etal-2025-vibe,
    title = "{V}i{B}e: A Text-to-Video Benchmark for Evaluating Hallucination in Large Multimodal Models",
    author = "Rawte, Vipula  and
      Jain, Sarthak  and
      Sinha, Aarush  and
      Kaushik, Garv  and
      Bansal, Aman  and
      Vishwanath, Prathiksha Rumale  and
      Jain, Samyak Rajesh  and
      Reganti, Aishwarya Naresh  and
      Jain, Vinija  and
      Chadha, Aman  and
      Sheth, Amit  and
      Das, Amitava",
    editor = "Cao, Trista  and
      Das, Anubrata  and
      Kumarage, Tharindu  and
      Wan, Yixin  and
      Krishna, Satyapriya  and
      Mehrabi, Ninareh  and
      Dhamala, Jwala  and
      Ramakrishna, Anil  and
      Galystan, Aram  and
      Kumar, Anoop  and
      Gupta, Rahul  and
      Chang, Kai-Wei",
    booktitle = "Proceedings of the 5th Workshop on Trustworthy NLP (TrustNLP 2025)",
    month = may,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.trustnlp-main.15/",
    pages = "232--246",
    ISBN = "979-8-89176-233-6",
    abstract = "Recent advances in Large Multimodal Models (LMMs) have expanded their capabilities to video understanding, with Text-to-Video (T2V) models excelling in generating videos from textual prompts. However, they still frequently produce hallucinated content, revealing AI-generated inconsistencies. We introduce ViBe \url{https://huggingface.co/datasets/ViBe-T2V-Bench/ViBe}: a large-scale dataset of hallucinated videos from open-source T2V models. We identify five major hallucination types: Vanishing Subject, Omission Error, Numeric Variability, Subject Dysmorphia, and Visual Incongruity. Using ten T2V models, we generated and manually annotated 3,782 videos from 837 diverse MS COCO captions. Our proposed benchmark includes a dataset of hallucinated videos and a classification framework using video embeddings. ViBe serves as a critical resource for evaluating T2V reliability and advancing hallucination detection. We establish classification as a baseline, with the TimeSFormer + CNN ensemble achieving the best performance (0.345 accuracy, 0.342 F1 score). While initial baselines proposed achieve modest accuracy, this highlights the difficulty of automated hallucination detection and the need for improved methods. Our research aims to drive the development of more robust T2V models and evaluate their outputs based on user preferences. Our code is available at: \url{https://anonymous.4open.science/r/vibe-1840/}"
}