@inproceedings{sun-etal-2025-dreamsync,
    title = "{D}ream{S}ync: Aligning Text-to-Image Generation with Image Understanding Feedback",
    author = "Sun, Jiao  and
      Fu, Deqing  and
      Hu, Yushi  and
      Wang, Su  and
      Rassin, Royi  and
      Juan, Da-Cheng  and
      Alon, Dana  and
      Herrmann, Charles  and
      Steenkiste, Sjoerd Van  and
      Krishna, Ranjay  and
      Rashtchian, Cyrus",
    editor = "Chiruzzo, Luis  and
      Ritter, Alan  and
      Wang, Lu",
    booktitle = "Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)",
    month = apr,
    year = "2025",
    address = "Albuquerque, New Mexico",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/fix-sig-urls/2025.naacl-long.304/",
    pages = "5920--5945",
    ISBN = "979-8-89176-189-6",
    abstract = "Despite their widespread success, Text-to-Image models (T2I) still struggle to produce images that are both aesthetically pleasing and faithful to the user{'}s input text. We introduce DreamSync, a simple yet effective training algorithm that improves T2I models to be faithful to the text input. DreamSync utilizes large vision-language models (VLMs) to effectively identify the fine-grained discrepancies between generated images and the text inputs and enable T2I models to self-improve without labeled data. First, it prompts the model to generate several candidate images for a given input text. Then, it uses two VLMs to select the best generation: a Visual Question Answering model that measures the alignment of generated images to the text, and another that measures the generation{'}s aesthetic quality. After selection, we use LoRA to iteratively finetune the T2I model to guide its generation towards the selected best generations. DreamSync does not need any additional human annotation, model architecture changes, or reinforcement learning. Despite its simplicity, DreamSync improves both the semantic alignment and aesthetic appeal of two diffusion-based T2I models, evidenced by multiple benchmarks (+1.7{\%} on TIFA, +2.9{\%} on DSG1K, +3.4{\%} on VILA aesthetic) and human evaluation shows that DreamSync improves text rendering compared to SDXL by 18.5{\%} on DSG1K benchmark."
}