Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Models
Umberto Cappellazzo, Xubo Liu, Pingchuan Ma, Stavros Petridis, Maja Pantic
2025-11-11
Summary
This paper introduces a new system called Omni-AVSR, which is a single artificial intelligence model designed to understand speech from both audio and video sources, and even when only audio or video is available.
What's the problem?
Current AI systems for understanding speech, like those that convert speech to text, usually train separate models for audio, video (lip reading), and combined audio-video input. This is inefficient because it requires a lot of computing power and storage space for each model, and it doesn't allow the models to learn from each other. Also, these systems often compress the audio and video data in a rigid way, limiting how well they can balance accuracy and speed.
What's the solution?
The researchers created Omni-AVSR, a single model that can handle all three speech recognition tasks (audio, video, and combined). They used a technique called 'matryoshka representation learning' to train the model efficiently on different levels of detail in the audio and video. They also used a method called 'LoRA' to fine-tune the model for each specific task without retraining the entire system, making it adaptable and resource-friendly.
Why it matters?
This work is important because it shows that a single AI model can perform multiple speech recognition tasks just as well as, or even better than, separate models. This reduces the computational cost and resources needed for speech recognition, making it more practical for real-world applications and allowing for faster processing, especially in noisy environments. It also helps us understand how to best build and scale these types of models in the future.
Abstract
Large language models (LLMs) have recently achieved impressive results in speech recognition across multiple modalities, including Auditory Speech Recognition (ASR), Visual Speech Recognition (VSR), and Audio-Visual Speech Recognition (AVSR). Despite this progress, current LLM-based approaches typically address each task independently, training separate models that raise computational and deployment resource use while missing potential cross-task synergies. They also rely on fixed-rate token compression, which restricts flexibility in balancing accuracy with efficiency. These limitations highlight the need for a unified framework that can support ASR, VSR, and AVSR while enabling elastic inference. To this end, we present Omni-AVSR, a unified audio-visual LLM that combines efficient multi-granularity training with parameter-efficient adaptation. Specifically, we adapt the matryoshka representation learning paradigm to efficiently train across multiple audio and visual granularities, reducing its inherent training resource use. Furthermore, we explore three LoRA-based strategies for adapting the backbone LLM, balancing shared and task-specific specialization. Experiments on LRS2 and LRS3 show that Omni-AVSR achieves comparable or superior accuracy to state-of-the-art baselines while training a single model at substantially lower training and deployment resource use. The model also remains robust under acoustic noise, and we analyze its scaling behavior as LLM size increases, providing insights into the trade-off between performance and efficiency.