Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts
Weigao Sun, Disen Lan, Tong Zhu, Xiaoye Qu, Yu Cheng
2025-03-10
Summary
This paper talks about Linear-MoE, a new AI system that combines two advanced techniques, Linear Sequence Modeling (LSM) and Mixture-of-Experts (MoE), to make large-scale models faster and more efficient
What's the problem?
Large AI models are powerful but often require a lot of computing resources and time to train. Existing methods struggle to balance efficiency with performance, especially when handling long sequences of data
What's the solution?
The researchers created Linear-MoE, which uses LSM for simpler and faster sequence processing and MoE to activate only specific parts of the model when needed. They also developed a system with two subsystems: one for modeling different types of LSM methods and another for efficient training using advanced techniques like Sequence Parallelism. They tested Linear-MoE on various benchmarks and showed it performs well while using fewer resources
Why it matters?
This matters because Linear-MoE makes it possible to train large AI models more efficiently, saving time and computational power. It could help create better AI systems for tasks like language understanding, coding, or science without requiring expensive hardware, making advanced AI tools more accessible
Abstract
Linear Sequence Modeling (LSM) like linear attention, state space models and linear RNNs, and Mixture-of-Experts (MoE) have recently emerged as significant architectural improvements. In this paper, we introduce Linear-MoE, a production-level system for modeling and training large-scale models that integrate LSM with MoE. Linear-MoE leverages the advantages of both LSM modules for linear-complexity sequence modeling and MoE layers for sparsely activation, aiming to offer high performance with efficient training. The Linear-MoE system comprises: 1) Modeling subsystem, which provides a unified framework supporting all instances of LSM. and 2) Training subsystem, which facilitates efficient training by incorporating various advanced parallelism technologies, particularly Sequence Parallelism designed for Linear-MoE models. Additionally, we explore hybrid models that combine Linear-MoE layers with standard Transformer-MoE layers with its Sequence Parallelism to further enhance model flexibility and performance. Evaluations on two model series, A0.3B-2B and A1B-7B, demonstrate Linear-MoE achieves efficiency gains while maintaining competitive performance on various benchmarks, showcasing its potential as a next-generation foundational model architecture. Code: https://github.com/OpenSparseLLMs/Linear-MoE.