Dekun Chen
2026
MimicLM: Zero-Shot Voice Imitation through Autoregressive Modeling of Pseudo-Parallel Speech Corpora
Tao Feng | Yuxiang Wang | Yuancheng Wang | Xueyao Zhang | Dekun Chen | Chaoren Wang | Xun Guan | Zhizheng Wu
Findings of the Association for Computational Linguistics: ACL 2026
Tao Feng | Yuxiang Wang | Yuancheng Wang | Xueyao Zhang | Dekun Chen | Chaoren Wang | Xun Guan | Zhizheng Wu
Findings of the Association for Computational Linguistics: ACL 2026
Voice imitation aims to transform *source* speech to match a *reference* speaker’s timbre and speaking style while preserving linguistic content. A straightforward approach is to train on triplets of *(source, reference, target)*, where *source* and *target* share the same content but *target* matches the *reference*’s voice characteristics, yet such data is extremely scarce. Existing approaches either employ carefully designed disentanglement architectures to bypass this data scarcity or leverage external systems to synthesize pseudo-parallel training data. However, the former requires intricate model design, and the latter faces a quality ceiling when synthetic speech is used as training *targets*. To address these limitations, we propose MimicLM, which takes a novel approach by using synthetic speech as training *sources* while retaining real recordings as *targets*. This design enables the model to learn directly from real speech distributions, breaking the synthetic quality ceiling. Building on this data construction approach, we incorporate interleaved text-audio modeling to guide the generation of content-accurate speech and apply post-training with preference alignment to mitigate the inherent distributional mismatch when training on synthetic data. Experiments demonstrate that MimicLM achieves superior voice imitation quality with a simple yet effective architecture, significantly outperforming existing methods in naturalness while maintaining competitive similarity scores across speaker identity, accent, and emotion dimensions.