@inproceedings{veluri-etal-2024-beyond,
    title = "Beyond Turn-Based Interfaces: Synchronous {LLM}s as Full-Duplex Dialogue Agents",
    author = "Veluri, Bandhav  and
      Peloquin, Benjamin N  and
      Yu, Bokai  and
      Gong, Hongyu  and
      Gollakota, Shyamnath",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/moar-dois/2024.emnlp-main.1192/",
    doi = "10.18653/v1/2024.emnlp-main.1192",
    pages = "21390--21402",
    abstract = "Despite broad interest in modeling spoken dialogue agents, most approaches are inherently ``half-duplex'' {--} restricted to turn-based interaction with responses requiring explicit prompting by the user or implicit tracking of interruption or silence events. Human dialogue, by contrast, is ``full-duplex'' allowing for rich synchronicity in the form of quick and dynamic turn-taking, overlapping speech, and backchanneling. Technically, the challenge of achieving full-duplex dialogue with LLMs lies in modeling synchrony as pre-trained LLMs do not have a sense of ``time''. To bridge this gap, we propose Synchronous LLMs for full-duplex spoken dialogue modeling. We design a novel mechanism to integrate time information into Llama3-8b so that they run synchronously with the real-world clock. We also introduce a training recipe that uses 212k hours of synthetic spoken dialogue data generated from text dialogue data to create a model that generates meaningful and natural spoken dialogue, with just 2k hours of real-world spoken dialogue data. Synchronous LLMs outperform state-of-the-art in dialogue meaningfulness while maintaining naturalness. Finally, we demonstrate the model{'}s ability to participate in full-duplex dialogue by simulating interaction between two agents trained on different datasets, while considering Internet-scale latencies of up to 240 ms."
}