Navigating Connected Memories with a Task-oriented Dialog System
Satwik Kottur, Seungwhan Moon, Alborz Geramifard, Babak Damavandi
Abstract
Recent years have seen an increasing trend in the volume of personal media captured by users, thanks to the advent of smartphones and smart glasses, resulting in large media collections. Despite conversation being an intuitive human-computer interface, current efforts focus mostly on single-shot natural language based media retrieval to aid users query their media and re-live their memories. This severely limits the search functionality as users can neither ask follow-up queries nor obtain information without first formulating a single-turn query.In this work, we propose dialogs for connected memories as a powerful tool to empower users to search their media collection through a multi-turn, interactive conversation. Towards this, we collect a new task-oriented dialog dataset COMET, which contains 11.5k user↔assistant dialogs (totalling 103k utterances), grounded in simulated personal memory graphs. We employ a resource-efficient, two-phase data collection pipeline that uses: (1) a novel multimodal dialog simulator that generates synthetic dialog flows grounded in memory graphs, and, (2) manual paraphrasing to obtain natural language utterances. We analyze COMET, formulate four main tasks to benchmark meaningful progress, and adopt state-of-the-art language models as strong baselines, in order to highlight the multimodal challenges captured by our dataset.- Anthology ID:
- 2022.emnlp-main.160
- Volume:
- Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing
- Month:
- December
- Year:
- 2022
- Address:
- Abu Dhabi, United Arab Emirates
- Editors:
- Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
- Venue:
- EMNLP
- SIG:
- Publisher:
- Association for Computational Linguistics
- Note:
- Pages:
- 2495–2507
- Language:
- URL:
- https://aclanthology.org/2022.emnlp-main.160
- DOI:
- 10.18653/v1/2022.emnlp-main.160
- Cite (ACL):
- Satwik Kottur, Seungwhan Moon, Alborz Geramifard, and Babak Damavandi. 2022. Navigating Connected Memories with a Task-oriented Dialog System. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2495–2507, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
- Cite (Informal):
- Navigating Connected Memories with a Task-oriented Dialog System (Kottur et al., EMNLP 2022)
- PDF:
- https://preview.aclanthology.org/add_acl24_videos/2022.emnlp-main.160.pdf