Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMs
Umberto Cappellazzo, Minsu Kim, Stavros Petridis
2025-03-11
Summary
This paper talks about Llama-MTSK, an AI tool that combines sound and lip-reading to understand speech better in noisy places, using a smart 'nested' design to save computer power without losing accuracy.
What's the problem?
Current speech AI tools that use both audio and video need too much computing power when paired with large language models, forcing a choice between speed and accuracy.
What's the solution?
Llama-MTSK uses a 'Matryoshka doll' method to process speech data at different levels of detail in one model, and adds lightweight add-ons (LoRA) to tweak the AI efficiently, keeping it fast and accurate.
Why it matters?
This helps devices like smart assistants and hearing aids understand speech reliably in loud environments while using less battery or processing power.
Abstract
Audio-Visual Speech Recognition (AVSR) leverages both audio and visual modalities to enhance speech recognition robustness, particularly in noisy environments. Recent advancements in Large Language Models (LLMs) have demonstrated their effectiveness in speech recognition, including AVSR. However, due to the significant length of speech representations, direct integration with LLMs imposes substantial computational costs. Prior approaches address this by compressing speech representations before feeding them into LLMs. However, higher compression ratios often lead to performance degradation, necessitating a trade-off between computational efficiency and recognition accuracy. To address this challenge, we propose Llama-MTSK, the first Matryoshka-based Multimodal LLM for AVSR, which enables flexible adaptation of the audio-visual token allocation based on specific computational constraints while preserving high performance. Our approach, inspired by Matryoshka Representation Learning, encodes audio-visual representations at multiple granularities within a single model, eliminating the need to train separate models for different compression levels. Moreover, to efficiently fine-tune the LLM, we introduce three LoRA-based Matryoshka strategies using global and scale-specific LoRA modules. Extensive evaluations on the two largest AVSR datasets demonstrate that Llama-MTSK achieves state-of-the-art results, matching or surpassing models trained independently at fixed compression levels.