@inproceedings{osebe-etal-2024-towards,
    title = "Towards Multi-Modal Co-Reference Resolution in Conversational Shopping Agents",
    author = "Osebe, Samuel  and
      Wanigasekara, Prashan  and
      Gueudre, Thomas  and
      Tran, Thanh  and
      Sharma, Rahul  and
      Yang, Fan  and
      Hu, Qian  and
      Ruan, Weitong  and
      Barut, Emre  and
      Su, Chengwei",
    editor = "Malmasi, Shervin  and
      Fetahu, Besnik  and
      Ueffing, Nicola  and
      Rokhlenko, Oleg  and
      Agichtein, Eugene  and
      Guy, Ido",
    booktitle = "Proceedings of the Seventh Workshop on e-Commerce and NLP @ LREC-COLING 2024",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.ecnlp-1.2",
    pages = "8--18",
    abstract = "The context of modern smart voice assistants is often multi-modal, where images, audio and video content are consumed by users simultaneously. In such a setup, co-reference resolution is especially challenging, and runs across modalities and dialogue turns. We explore the problem of multi-modal co-reference resolution in multi-turn dialogues and quantify the performance of multi-modal LLMs on a specially curated dataset of long, image-interleaved conversations between a voice assistant and human in a shopping use case. We propose a custom architecture for multi-modal embedding alignment using a novel parameter augmentation technique. Our proposed Parameter Augmented LLM approach shows a 4.9{\%} absolute F1 improvement above a cross-attention baseline while reducing the number of parameters being trained by 4x.",
}