Collaborative Research: RI: Medium: Using Systematic Relationships and Phonetic Speech Foundation Models for Universal Speech-to-Text across Varieties of Languages

About This Grant

This project introduces a new way to develop speech-to-text systems for language varieties in which little digital data is available, especially when a related variety already has plenty of digital data. Languages differ from one another, but they also show a great deal of variation within themselves. People in different regions or countries often pronounce words in the same language differently. For example, many people in the southern US pronounce the “i” in words like “ride” as an “a” sound (similar to the one found in “rad”), while most other Americans do not. Most speech-to-text systems—which turn spoken words into written text—focus on just one variety of each language. However, in everyday life, many people speak other varieties. Existing speech-to-text systems often do not work well for these varieties. Improving speech recognition for low-resource varieties represents a business opportunity, one that can help more Americans access voice-powered tools and services. This project takes one step towards this goal. It innovates by leveraging the fact that the differences between various varieties of the same language often follow predictable patterns. For example, since the pronunciations of words in different regions change following rules that apply to the whole vocabulary, one can often predict how a word will be pronounced in one variety if one knows how it is pronounced in another. The project will develop a powerful AI model (POWSM) that can transcribe pronunciation (using a universal system developed by linguists to represent sounds). This approach enables the development of speech-to-text systems for previously unsupported language variants, even when little recorded training data exists for them. It works by learning both the similarities and the systematic differences between well-resourced varieties and others. The project builds an encoder-decoder foundation speech model called POWSM (Phonetic Open Whisper-style Speech Model), which is trained to recognize speech as sequences of phones (consonants and vowels) in any language. The project will include three applications using this model: 1) Prompting POWSM with vector representations of the systematic sound correspondences between a low-resource variety (LRV) and a high-resource variety (HRV), enabling the model to recognize the LRV as a transformed variant of the HRV. 2) Constructing stochastic weighted finite-state transducers that can generate synthetic LRV data based on linguist-curated knowledge about sound changes in the HRV and LRV. Using this synthetic data to train a language model that can be used to decode LRV output from POWSM. 3) Learning phonetic correspondences between language varieties automatically from transcribed audio using a novel form of unsupervised bilingual lexicon induction (UBLI) that leverages both text and audio. Audio aligned in this way can be used to train basic speech-to-speech translation models that, when used in conjunction with POWSM, enable speech technologies for LRVs without requiring linguistic annotation beyond transcription. The proposed methods are intended to cover most HRV-LRV scenarios. They will be evaluated on major languages including varieties in English, Italian, Chinese, German, Arabic, German, and Dutch, English, as well as endangered languages like Nahuatl and Mixtec. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.