Maike Z\"ufle
2026
F-Actor: Controllable Conversational Behavior in Full-Duplex Models
Maike Z\"ufle | Ondrej Klejch | Nicholas Sanders | Jan Niehues | Alexandra Birch | Tsz Kin Lam
Findings of the Association for Computational Linguistics: ACL 2026
Maike Z\"ufle | Ondrej Klejch | Nicholas Sanders | Jan Niehues | Alexandra Birch | Tsz Kin Lam
Findings of the Association for Computational Linguistics: ACL 2026
Spoken conversational systems require more than accurate speech generation to have human-like conversations: to feel natural and engaging, they must produce conversational behaviour that adapts dynamically to the context. Current spoken conversational systems, however, rarely allow such customization, limiting their naturalness and usability. In this work, we present the first open, instruction-following full-duplex conversational speech model that can be trained efficiently under typical academic resource constraints. By keeping the audio encoder frozen and finetuning only the language model, our model requires just 2,000 hours of data, without relying on large-scale pretraining or multi-stage optimization. The model can follow explicit instructions to control speaker voice, conversation topic, conversational behaviour (e.g., backchanneling and interruptions), and dialogue initiation. We propose a single-stage training protocol and systematically analyze design choices. Both the model and training code is released to enable reproducible research on controllable full-duplex speech systems.
Beyond Transcripts: A Renewed Perspective on Audio Chaptering
Fabian Retkowski | Maike Z\"ufle | Thai Binh Nguyen | Jan Niehues | Alexander Waibel
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Fabian Retkowski | Maike Z\"ufle | Thai Binh Nguyen | Jan Niehues | Alexander Waibel
Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
Audio chaptering, the task of automatically segmenting long-form audio into coherent sections, is increasingly important for navigating podcasts, lectures, and videos. Despite its relevance, research remains limited and text-based, leaving key questions unresolved about leveraging audio information, handling ASR errors, and transcript-free evaluation. We address these gaps through three contributions: (1) a systematic comparison between text-based models with acoustic features, a novel audio-only architecture (AudioSeg) operating on learned audio representations, and multimodal LLMs; (2) empirical analysis of factors affecting performance, including transcript quality, acoustic features, duration, and speaker composition; and (3) formalized evaluation protocols contrasting transcript-dependent text-space protocols with transcript-invariant time-space protocols. Our experiments on YTSeg reveal that AudioSeg substantially outperforms text-based approaches, pauses provide the largest acoustic gains, and current MLLMs struggle due to context limitations and weak instruction following.