MulliVC: Multi-lingual Voice Conversion With Cycle Consistency
Jiawei Huang, Chen Zhang, Yi Ren, Ziyue Jiang, Zhenhui Ye, Jinglin Liu, Jinzheng He, Xiang Yin, Zhou Zhao
2024-08-12
Summary
This paper presents MulliVC, a new system for converting voices across different languages while keeping the original speech content intact.
What's the problem?
Voice conversion technology aims to change one person's voice to sound like another's while maintaining what is being said. However, doing this across multiple languages is challenging due to differences in how people speak and the lack of available data that pairs voices from the same speaker in different languages. This makes it hard to create models that can effectively convert voices while preserving the original meaning and tone.
What's the solution?
MulliVC tackles these challenges by using a unique approach that focuses on converting only the timbre (the quality of the voice) without needing paired multilingual data. The system is trained in three steps: first, it learns from monolingual speech data; second, it uses a cyclical process inspired by back translation to separate the voice characteristics from other information like content and prosody (the rhythm and intonation of speech). This allows MulliVC to effectively convert voices even when there is no direct multilingual data available.
Why it matters?
This research is significant because it advances the field of voice conversion, making it possible to create applications like dubbing movies or translating speeches into different languages without losing the original speaker's identity. By overcoming the limitations of existing methods, MulliVC opens up new possibilities for communication and media in a multilingual world.
Abstract
Voice conversion aims to modify the source speaker's voice to resemble the target speaker while preserving the original speech content. Despite notable advancements in voice conversion these days, multi-lingual voice conversion (including both monolingual and cross-lingual scenarios) has yet to be extensively studied. It faces two main challenges: 1) the considerable variability in prosody and articulation habits across languages; and 2) the rarity of paired multi-lingual datasets from the same speaker. In this paper, we propose MulliVC, a novel voice conversion system that only converts timbre and keeps original content and source language prosody without multi-lingual paired data. Specifically, each training step of MulliVC contains three substeps: In step one the model is trained with monolingual speech data; then, steps two and three take inspiration from back translation, construct a cyclical process to disentangle the timbre and other information (content, prosody, and other language-related information) in the absence of multi-lingual data from the same speaker. Both objective and subjective results indicate that MulliVC significantly surpasses other methods in both monolingual and cross-lingual contexts, demonstrating the system's efficacy and the viability of the three-step approach with cycle consistency. Audio samples can be found on our demo page (mullivc.github.io).