@inproceedings{huang-etal-2025-enhancing-multimodal,
    title = "Enhancing Multimodal Unified Representations for Cross Modal Generalization",
    author = "Huang, Hai  and
      Xia, Yan  and
      Ji, Shengpeng  and
      Wang, Shulei  and
      Wang, Hanting  and
      Fang, Minghui  and
      Zhu, Jieming  and
      Dong, Zhenhua  and
      Zhou, Sashuai  and
      Zhao, Zhou",
    editor = "Che, Wanxiang  and
      Nabende, Joyce  and
      Shutova, Ekaterina  and
      Pilehvar, Mohammad Taher",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2025",
    month = jul,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/ingestion-acl-25/2025.findings-acl.119/",
    pages = "2353--2366",
    ISBN = "979-8-89176-256-5",
    abstract = "To enhance the interpretability of multimodal unified representations, many studies have focused on discrete unified representations. These efforts typically start with contrastive learning and gradually extend to the disentanglement of modal information, achieving solid multimodal discrete unified representations. However, existing research often overlooks two critical issues: 1) The use of Euclidean distance for quantization in discrete representations often overlooks the important distinctions among different dimensions of features, resulting in redundant representations after quantization; 2) Different modalities have unique characteristics, and a uniform alignment approach does not fully exploit these traits. To address these issues, we propose Training-free Optimization of Codebook (TOC) and Fine and Coarse cross-modal Information Disentangling (FCID). These methods refine the unified discrete representations from pretraining and perform fine- and coarse-grained information disentanglement tailored to the specific characteristics of each modality, achieving significant performance improvements over previous state-of-the-art models. The code is available at https://github.com/haihuangcode/CMG."
}