✦
Summary
HeartMuLa represents a groundbreaking family of open-source music foundation models designed to unify music generation and understanding through advanced components like HeartCLAP for audio-text alignment, HeartTranscriptor for robust lyric recognition in real-world scenarios, and HeartCodec for high-fidelity, low-frame-rate music tokenization that preserves long-range musical structures alongside fine-grained acoustic details. This framework enables efficient autoregressive modeling, allowing creators to produce professional, studio-quality tracks from diverse inputs. By operating on discrete audio tokens, HeartMuLa delivers long-form music synthesis up to six minutes in length, maintaining structural coherence and expressive diversity throughout extended compositions.
Key Features
Generates full songs up to six minutes from text prompts, lyrics, or reference audio
Fine-grained style control for song sections like intro, verse, chorus, and bridge
Multilingual support including English, Chinese, Japanese, Korean, and Spanish
High-fidelity audio codec at 12.5 Hz capturing long-range structure and details
Short-music mode for video background tracks
Audio2Music for creating song covers from reference audio
Controllable attributes like genre, mood, rhythm, and expressiveness
Local, open-source operation compatible with consumer hardware like AMD GPUs
About HeartMuLa
The core HeartMuLa model functions as an LLM-based song generator that accepts flexible, user-controllable conditions such as textual style descriptions, detailed lyrics, and reference audio inputs. It excels in multilingual support, covering languages like English, Chinese, Japanese, Korean, and Spanish, making it accessible for global creators. Specialized modes enhance its versatility: fine-grained musical attribute control lets users specify styles for individual song sections like intros, verses, choruses, and bridges using natural language prompts, while a short-music generation mode produces engaging clips ideal for background tracks in videos.
Built for both research and creative pipelines, HeartMuLa supports detailed control over elements like genre, mood, rhythm, and expressive variations, positioning it as a powerful tool for music production. Its hierarchical architecture ensures high-fidelity output even in local environments, with demonstrations showing competitive performance against proprietary systems. As an open-source solution under Apache 2.0, it fosters innovation in AI-driven music, enabling covers via Audio2Music functionality and encouraging community development for non-commercial and now commercially viable applications.