@inproceedings{nayak-etal-2025-culturalframes,
    title = "{C}ultural{F}rames: Assessing Cultural Expectation Alignment in Text-to-Image Models and Evaluation Metrics",
    author = "Nayak, Shravan  and
      Bhatia, Mehar  and
      Zhang, Xiaofeng  and
      Rieser, Verena  and
      Hendricks, Lisa Anne  and
      Steenkiste, Sjoerd Van  and
      Goyal, Yash  and
      Stanczak, Karolina  and
      Agrawal, Aishwarya",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/author-page-yu-wang-polytechnic/2025.findings-emnlp.1141/",
    doi = "10.18653/v1/2025.findings-emnlp.1141",
    pages = "20918--20953",
    ISBN = "979-8-89176-335-7",
    abstract = "The increasing ubiquity of text-to-image (T2I) models as tools for visual content generation raises concerns about their ability to accurately represent diverse cultural contexts - where missed cues can stereotype communities and undermine usability. In this work, we present the first study to systematically quantify the alignment of T2I models and evaluation metrics with respect to both explicit (stated) as well as implicit (unstated, implied by the prompt{'}s cultural context) cultural expectations. To this end, we introduce CulturalFrames, a novel benchmark designed for rigorous human evaluation of cultural representation in visual generations. Spanning 10 countries and 5 socio-cultural domains, CulturalFrames comprises 983 prompts, 3637 corresponding images generated by 4 state-of-the-art T2I models, and over 10k detailed human annotations. We find that across models and countries, cultural expectations are missed an average of 44{\%} of the time. Among these failures, explicit expectations are missed at a surprisingly high average rate of 68{\%}, while implicit expectation failures are also significant, averaging 49{\%}. Furthermore, we show that existing T2I evaluation metrics correlate poorly with human judgments of cultural alignment, irrespective of their internal reasoning. Collectively, our findings expose critical gaps, provide a concrete testbed, and outline actionable directions for developing culturally informed T2I models and metrics that improve global usability."
}