@inproceedings{kumar-etal-2024-tokenverse,
    title = "{T}oken{V}erse: Towards Unifying Speech and {NLP} Tasks via Transducer-based {ASR}",
    author = "Kumar, Shashi  and
      Madikeri, Srikanth  and
      Zuluaga Gomez, Juan Pablo  and
      Thorbecke, Iuliia  and
      Villatoro-tello, Esa{\'u}  and
      Burdisso, Sergio  and
      Motlicek, Petr  and
      S, Karthik Pandia D  and
      Ganapathiraju, Aravind",
    editor = "Al-Onaizan, Yaser  and
      Bansal, Mohit  and
      Chen, Yun-Nung",
    booktitle = "Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2024",
    address = "Miami, Florida, USA",
    publisher = "Association for Computational Linguistics",
    url = "https://preview.aclanthology.org/jlcl-multiple-ingestion/2024.emnlp-main.1167/",
    doi = "10.18653/v1/2024.emnlp-main.1167",
    pages = "20988--20995",
    abstract = "In traditional conversational intelligence from speech, a cascaded pipeline is used, involving tasks such as voice activity detection, diarization, transcription, and subsequent processing with different NLP models for tasks like semantic endpointing and named entity recognition (NER). Our paper introduces TokenVerse, a single Transducer-based model designed to handle multiple tasks. This is achieved by integrating task-specific tokens into the reference text during ASR model training, streamlining the inference and eliminating the need for separate NLP models. In addition to ASR, we conduct experiments on 3 different tasks: speaker change detection, endpointing, and NER. Our experiments on a public and a private dataset show that the proposed method improves ASR by up to 7.7{\%} in relative WER while outperforming the cascaded pipeline approach in individual task performance. Our code is publicly available: https://github.com/idiap/tokenverse-unifying-speech-nlp"
}