@inproceedings{drushchak-etal-2025-multimodal,
    title = "Multimodal Retrieval-Augmented Generation: Unified Information Processing Across Text, Image, Table, and Video Modalities",
    author = "Drushchak, Nazarii  and
      Polyakovska, Nataliya  and
      Bautina, Maryna  and
      Semenchenko, Taras  and
      Koscielecki, Jakub  and
      Sykala, Wojciech  and
      Wegrzynowski, Michal",
    editor = "Kriz, Reno  and
      Murray, Kenton",
    booktitle = "Proceedings of the 1st Workshop on Multimodal Augmented Generation via Multimodal Retrieval (MAGMaR 2025)",
    month = aug,
    year = "2025",
    address = "Vienna, Austria",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/landing_page/2025.magmar-1.5/",
    pages = "59--64",
    ISBN = "979-8-89176-280-0",
    abstract = "Retrieval-augmented generation (RAG) is a powerful paradigm for leveraging external data to enhance the capabilities of large language models (LLMs). However, most existing RAG solutions are tailored for single-modality or limited multimodal scenarios, restricting their applicability in real-world contexts where diverse data sources{---}including text, tables, images, and videos{---}must be integrated seamlessly. In this work proposes a unified \textit{Multimodal Retrieval-augmented generation (mRAG)} system designed to unify information processing across all four modalities. Our pipeline ingests and indexes data from PDFs and videos using tools like Amazon Textract, Transcribe, Langfuse, and multimodal LLMs (e.g., Claude 3.5 Sonnet) for structured extraction and semantic enrichment. The dataset includes text queries, table lookups, image-based questions, and videos. Evaluation with the Deepeval framework shows improved retrieval accuracy and response quality, especially for structured text and tables. While performance on image and video queries is lower, the multimodal integration framework remains robust, underscoring the value of unified pipelines for diverse data."
}