@inproceedings{zhang-etal-2025-diffusion,
    title = "Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective",
    author = "Zhang, Siyue  and
      Zhao, Yilun  and
      Geng, Liyuan  and
      Cohan, Arman  and
      Luu, Anh Tuan  and
      Zhao, Chen",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingest-emnlp/2025.emnlp-main.213/",
    pages = "4273--4303",
    ISBN = "979-8-89176-332-6",
    abstract = "Large language model (LLM)-based embedding models, benefiting from large scale pre-training and post-training, have begun to surpass BERT and T5-based models on general-purpose text embedding tasks such as document retrieval. However, a fundamental limitation of LLM embeddings lies in the unidirectional attention used during autoregressive pre-training, which misaligns with the bidirectional nature of text embedding tasks. To this end, We propose adopting diffusion language models for text embeddings, motivated by their inherent bidirectional architecture and recent success in matching or surpassing LLMs especially on reasoning tasks. We present the first systematic study of the diffusion language embedding model, which outperforms the LLM-based embedding model by 20{\%} on long-document retrieval, 8{\%} on reasoning-intensive retrieval, 2{\%} on instruction-following retrieval, and achieve competitive performance on traditional text embedding benchmarks. Our analysis verifies that bidirectional attention is crucial for encoding global context in long and complex text."
}