Style transfer is essential for high-quality machine dubbing. While numerous approaches for style transfer in speech synthesis have been developed, cross-lingual style transfer remains a significant challenge. In this paper we introduce a novel speech synthesis method which realizes style learning across different languages and speakers. Our approach features a transformer-based architecture with a speech prompted text encoder, a duration predictor and a flow matching generative decoder. The text encoder is conditioned on the noisy source language speech, which is entered as speech prompt for style adaptation. The flow matching generative decoder produces high quality speech conditioned on the text encoder output. Empirical evaluations demonstrate that our TTS system generates speech that is objectively closer to the recordings of professional voice talents compared to a strong baseline model. Audio samples are available on our demo page.
| English transcript | Bulgarian transcript | Noisy original prompt | Human dubbed speech | Proposed model speech | Baseline model speech |
|---|---|---|---|---|---|
| It looks fine. this is going through. | Изглежда добре, преминава! | ||||
| Let's do this. ready? | Да действаме. Готови? | ||||
| What? Oh no. Eli's rattle. That sucks. | Какво? О, не… Дрънкалката на Илай. Много жалко. | ||||
| Here we go again. It's moving. | Започваме отново, движи се! | ||||
| As Bear alerts the wolfpack, the flames quickly spread. | Докато Беър известява глутницата, пламъците бързо се разпространяват. | ||||
| Is everybody accounted for here? We weren't gonna stop until we found y'all. | Всички ли са налице? Няма да спрем, докато не намерим всички. | ||||
| It's ok, it's ok. It's ok. | Всичко е наред, всичко е наред! Наред е! | ||||
| Yeah. We got you know. | Да, хванахме те! | ||||
| I think she remembered me. I think she somehow likes me more. | Мисля, че си ме спомни. Мисля, че някак ме харесва повече. … | ||||
| They all have such a bond together, you know? | Имат такава близост помежду си, знаеш ли? | ||||
| It looks the same. Just looks a little bit more tired. | Изглежда същата, само малко по-изморена. | ||||
| Feel like in the long run, this is all that matters that it made it. | В дългосрочен план най-важното е оцеляло. | ||||
| Every since we left Alaska, I've almost felt lost. | Откакто напуснахме Аляска, се чувствам изгубена. | ||||
| With you wash, take your socks off. | Не може, свали ги! | ||||
| OK this is the last of it | Така, тези са последните! | ||||
| Ahh. Noah, it smells like home. | Ноа, мирише домашно! |