@inproceedings{huang-etal-2025-t2i,
    title = "{T}2{I}-{F}actual{B}ench: Benchmarking the Factuality of Text-to-Image Models with Knowledge-Intensive Concepts",
    author = "Huang, Ziwei  and
      He, Wanggui  and
      Long, Quanyu  and
      Wang, Yandi  and
      Li, Haoyuan  and
      Yu, Zhelun  and
      Shu, Fangxun  and
      Dai, Weilong  and
      Jiang, Hao  and
      Wu, Fei  and
      Gan, Leilei",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.acl-long.1334/",
    pages = "27501--27524",
    ISBN = "979-8-89176-251-0",
    abstract = "Most existing studies on evaluating text-to-image (T2I) models primarily focus on evaluating text-image alignment, image quality, and object composition capabilities, with comparatively fewer studies addressing the evaluation of the factuality of the synthesized images, particularly when the images involve knowledge-intensive concepts. In this work, we present T2I-FactualBench{---}the largest benchmark to date in terms of the number of concepts and prompts specifically designed to evaluate the factuality of knowledge-intensive concept generation. T2I-FactualBench consists of a three-tiered knowledge-intensive text-to-image generation framework, ranging from the basic memorization of individual knowledge concepts to the more complex composition of multiple knowledge concepts. We further introduce a multi-round visual question answering (VQA)-based evaluation framework to assesses the factuality of three-tiered knowledge-intensive text-to-image generation tasks. Experiments on T2I-FactualBench indicate that current state-of-the-art (SOTA) T2I models still leave significant room for improvement. We release our datasets and code at https://github.com/Safeoffellow/T2I-FactualBench."
}