ElevenLabs Unveils AI Dubbing: Translating Video and Audio into 20 Languages

ElevenLabs, a year-old voice cloning and synthesis startup founded by former Google and Palantir employees, is revolutionizing the way we dub audio and video content. Introducing AI Dubbing, a groundbreaking product that can translate any speech, including long-form content, into over 20 different languages.

With AI Dubbing, language barriers are no longer an obstacle for content creators. Now, even smaller creators without the resources to hire manual translators can take their content global.

“We have tested and iterated this feature in collaboration with hundreds of content creators to dub their content and make it more accessible to wider audiences,” says Mati Staniszewski, CEO and co-founder of ElevenLabs. “We see huge potential for independent creatives – such as those creating video content and podcasts – all the way through to film and TV studios.”

ElevenLabs claims that AI Dubbing can deliver high-quality translated audio in minutes, while retaining the original voice of the speaker, complete with their emotions and intonation.

But ElevenLabs is not the only player in the AI-driven translation space. In this age of AI, many enterprises are exploring speech-to-speech translation to drive efficiencies.

AI Dubbing: How it works

Using AI Dubbing is simple. Users just need to select the AI Dubbing tool on ElevenLabs, create a new project, choose the source and target languages, and upload the content file. The tool takes care of the rest.

Behind the scenes, ElevenLabs’ proprietary method removes background noise, distinguishes music and noise from dialogue, transcribes the speech, and translates and adapts the text to match the desired speech. The result is a high-quality dubbed output that retains the original voice characteristics of the speaker.

ElevenLabs’ Multilingual v2 model supports over 20 languages, giving users a wide range of options to globalize their content.

Staniszewski confirms that the new dubbing feature will be available to all users of the platform, with some character limits. Approximately one minute of AI Dubbing equates to 3,000 characters.

AI-based voices are coming

While ElevenLabs is making waves with its AI Dubbing tool, it’s not the only company exploring AI-based voicing. OpenAI, backed by Microsoft, recently made ChatGPT multimodal, allowing for conversations in response to voice prompts.

Other players in the AI-powered voice and speech synthesis space include MURF.AI, Play.ht, and WellSaid Labs. Meta has also launched SeamlessM4T, an open-source multilingual foundational model for real-time translations.

The global market for AI-powered voice and speech synthesis tools is projected to reach nearly $5 billion by 2032, with a CAGR of over 15.40%.