POWSM: A Phonetic Open Whisper-Style Speech Foundation Model

Summary

This paper introduces a new model called POWSM that can handle multiple speech-related tasks at once, like converting speech to text, text to phonetic sounds, and phonetic sounds back to speech.

What's the problem?

Traditionally, different speech tasks like automatic speech recognition, figuring out the sounds in speech, and converting writing into sounds have been treated as separate problems, each needing its own specialized tools and data. This is inefficient and doesn't allow for sharing knowledge between these related areas.

What's the solution?

The researchers created POWSM, a single model that can perform all these phonetic tasks together. It's designed to seamlessly move between audio, written text, and the individual sounds that make up speech. They showed that POWSM works as well as, or even better than, models specifically designed for just one of these tasks.

Why it matters?

This is a big step towards creating more versatile and efficient speech processing systems. A unified model like POWSM is especially helpful when working with languages that don't have a lot of existing data, because it can learn from multiple tasks at once. The researchers also made their model and data publicly available, encouraging further research in this area.