Meta unveils speech-to-text, text-to-speech AI models for over 1,100 languages; even shares open source data

All the tech majors are in a fierce fight over delivering utility to users in the form of artificial intelligence (AI) boosted products. While everyone knows about OpenAI's ChatGPT and Google's Bard, there was very little available on it from Facebook co-founder Mark Zuckerberg's Meta Platforms. Till today, that is. Now, the company has launched its speech-to-text, text-to-speech AI models for over 1,100 languages and the best part is that it is not linked to ChatGPT. Check out the Massively Multilingual Speech (MMS) project.

The biggest takeaway is that Meta has shared the open source and that means it could lead to a skyrocketing of the number of speech apps created across the world.

If all goes well in the real world, how useful this can be is clear from Meta's statement, "Existing speech recognition models only cover approximately 100 languages — a fraction of the 7,000+ known languages spoken on the planet."

Data Crunching

Now, good machine-learning models require large amounts of labeled data — in this case, many thousands of hours of audio, along with transcriptions. For most languages, this data simply does not exist.

However, Meta has overcome that through its MMS project, which combined wav2vec 2.0, its pioneering work in self-supervised learning, and a new dataset that provides labeled data for over 1,100 languages and unlabeled data for nearly 4,000 languages.

Patting itself on the back, Meta, in a statement said, "Our results show that the Massively Multilingual Speech models outperform existing models and cover 10 times as many languages."

It also revealed that, "Today, we are publicly sharing our models and code so that others in the research community can build upon our work. Through this work, we hope to make a small contribution to preserve the incredible language diversity of the world."

How Meta did it

The MMS project's first job was to collect audio data for thousands of languages, but the largest existing speech datasets covered at most 100 languages. The challenge was overcome by "turning to religious texts, such as the Bible, that have been translated in many different languages and whose translations have been widely studied for text-based language translation research".

The MMS project even created a dataset of readings of the New Testament in over 1,100 languages.

Having sensed that the idea was good and that it could be milked for much more, the project also considered unlabeled recordings of various other Christian religious readings. This increased the number of languages available to over 4,000.

Bias, what bias?

EVen though the data is from a specific domain, the biases seemed not to have entered into the system. This is clear from the fact that even though this text is often read by male speakers, Meta analysis showed that its MMS models perform equally well for male and female voices.

And, importantly, though the content of the audio recordings is religious, MMS analysis shows that this does not overly bias the model to produce more religious language.

Meta credits this success to the use of the Connectionist Temporal Classification approach, which it found to be better than the large language models (LLMs) or sequence to-sequence models for speech recognition.

How it was made usable

Meta preprocessed the data to make it usable by machine learning algorithms by training an alignment model on existing data in over 100 languages.

To reduce the error rate, Meta said, "We applied multiple rounds of this process and performed a final cross-validation filtering step based on model accuracy to remove potentially misaligned data.

Results obtained

Meta trained multilingual speech recognition models on over 1,100 languages. The consequence of this was explained by Meta in this way, "As the number of languages increases, performance does decrease, but only very slightly: Moving from 61 to 1,107 languages increases the character error rate by only about 0.4 percent but increases the language coverage by over 18 times."

MMS vs OpenAI Whisper

In a like-for-like comparison with Whisper, Meta said that models trained on the Massively Multilingual Speech data achieve only half the word error rate, but importantly, Massively Multilingual Speech covers 11 times more languages.