@inproceedings{mao-etal-2025-cross,
    title = "Cross-Modal Learning for Music-to-Music-Video Description Generation",
    author = "Mao, Zhuoyuan  and
      Zhao, Mengjie  and
      Wu, Qiyu  and
      Zhong, Zhi  and
      Liao, Wei-Hsiang  and
      Wakaki, Hiromi  and
      Mitsufuji, Yuki",
    editor = "Adlakha, Vaibhav  and
      Chronopoulou, Alexandra  and
      Li, Xiang Lorraine  and
      Majumder, Bodhisattwa Prasad  and
      Shi, Freda  and
      Vernikos, Giorgos",
    booktitle = "Proceedings of the 10th Workshop on Representation Learning for NLP (RepL4NLP-2025)",
    month = may,
    year = "2025",
    address = "Albuquerque, NM",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/moar-dois/2025.repl4nlp-1.4/",
    doi = "10.18653/v1/2025.repl4nlp-1.4",
    pages = "51--58",
    ISBN = "979-8-89176-245-9",
    abstract = "Music-to-music-video generation is a challenging task due to the intrinsic differences between the music and video modalities. The advent of powerful text-to-video diffusion models has opened a promising pathway for music-video (MV) generation by first addressing the music-to-MV description task and subsequently leveraging these models for video generation. In this study, we focus on the MV description generation task and propose a comprehensive pipeline encompassing training data construction and multimodal model fine-tuning. We fine-tune existing pre-trained multimodal models on our newly constructed music-to-MV description dataset based on the Music4All dataset, which integrates both musical and visual information. Our experimental results demonstrate that music representations can be effectively mapped to textual domains, enabling the generation of meaningful MV description directly from music inputs. We also identify key components in the dataset construction pipeline that critically impact the quality of MV description and highlight specific musical attributes that warrant greater focus for improved MV description generation."
}