The Science Behind How Microsoft’s AI Can Mimic Anyone’s Voice in 3 Seconds

January 17, 2023 • 0 Comment

You may understand how powerful artificial intelligence (AI) programs are getting. They are mimicking the arts of greats and writing scripts for global leaders, causing global conversations surrounding the rapid expansion of AI tools.

Most recently, Microsoft developed an AI for voice mimicry. Its efficiency and accuracy are groundbreaking, yet users question its purpose and utilization. How does this tool function, and how does it replicate voices in such a short amount of time?

Meet VALL-E

VALL-E is the AI learning to replicate voices within three seconds. VALL-E is one of the first to be so quick to train, as previous iterations of text-to-speech (TTS) programs struggled with efficiency and the nuances of sounds.

“However, VALL-E improves the research for this study in every regard, reducing training time and increasing the accuracy of complex voice qualities like tone and pacing.”

One of VALL-E’s goals was to replicate details that previous TTS programs couldn’t, and trials show mixed results for the current model. For authenticity, a program like this can’t only copy the person’s voice — it has to replicate the recording device’s audio quality and background environmental influences like static or noise. Though researchers remain impressed by its replicative qualities, Microsoft seeks further improvements in honing timbre and emotion-informed inflections.

Because VALL-E isn’t publicly available yet, it’s unsure how it will perform on larger scales. Microsoft is taking its time, refining it before public release to ensure proper usage.

Since VALL-E operates with minimal voice sampling, it’s uncertain how well it will generate longer audio clips with cohesion. AI and machine learning datasets have nearly countless data points for consideration. It’s a leap into AI’s future if Microsoft perfects voice replication with such a small frame of reference.

Know the Science

VALL-E is successful in its purpose because it melds well with existing technology. For example, GPT-3 still uses language processing models to perfect its TTS generation abilities for clear production and accurate editing. However, other models manipulate their data sets to create new content. VALL-E makes original content.

In collaboration with Meta, Microsoft uses EnCodec and LibriLight to inform VALL-E. EnCodec is an audio compression neural net capable of discerning even the most minor changes in audio. LibriLight is an audio library containing over 60,000 hours of English-speaking files from various voices.

“With these powers, VALL-E can take the three-second audio clip, transform it into a token EnCodec can analyze, and reference it to the library’s data for producing genuine-sounding vocal replications. Because EnCodec generates files in low bitrates, generation is quicker than other models of its kind.”

A sequence like this will generate more natural-sounding audio clips, competent at tricking even the most trained ear or voice-recognition technology.

The potential for this to assist industries is unquantifiable. It could increase efficiency and productivity while alleviating stress in every sector, not just communications. However, it has an equal chance to exacerbate criminal activity in the digital space, alongside other consequences.

Participate in Conversation

As with most AI advancements, ethical concerns present themselves. As with any text generation, AI works from data — therefore, plagiarism will always be a consideration. However, VALL-E references copyright-free sources, so this isn’t a primary concern yet.

However, Microsoft must also be wary of the public using technology like this for hostile purposes, such as spreading fake news or muddling investigations with false testimonies — possibly from witnesses no longer living. Specific industries, like law, will have to invent new policies and structures on how to encounter deepfakes in the courtroom.

“As with any technological advancement, misuse isn’t just probable — it’s unavoidable.”

Apart from the creative property or identity theft threat, proficient voice-generating AI could threaten the livelihoods of some professions or remove artistic and professional expertise from industries previously reliant on years dedicated to a craft.

Voice actors, speechwriters, and customer service representatives could all become obsolete with AI voice mimicry. The possibility of this is unknown and likely not possible in a rapid, comprehensive way. The conceivable outcome is voice mimicry will supplement these industries instead of replacing them. AI voice generation could help with idea generation or act as another worker to delegate tasks, offloading from human workers.

Getting on Speaking Terms With Voice-Replicating AI

Despite ethical concerns with voice-replicating AI, Microsoft is innovating a progressive, resourceful tool for the new generation — depending on how the public uses it. The science behind this tool is the most revolutionary aspect, and it could inform engineers and developers how to expand and transform AI for future applications in all sectors.

Technology implemented with VALL-E could translate into mindset shifts for the industry. The collaborative nature of this project will progress AI interactivity and development into a new age of accuracy and efficiency.