Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models
Ivan Sedykh, Nikita Sorokin, Valentin Malykh
2026-04-14
Summary
This paper investigates ways to speed up the process of generating text using a new type of language model called masked diffusion language models, which are becoming competitive with more traditional models but are currently slow to use.
What's the problem?
Masked diffusion language models work by gradually removing noise from data to create text, and this requires many processing steps. Each step needs a powerful computer processor, and unlike other text generation methods, it can't use shortcuts to remember what it's already generated, making it slow and computationally expensive. The core issue is that all steps in the denoising process aren't equally important, and some can be skipped or done with less powerful resources without significantly impacting the final text quality.
What's the solution?
The researchers found that the beginning and end stages of the denoising process are more forgiving and can be handled by a smaller, less powerful version of the model. They tested replacing the full model with a smaller one during certain steps, specifically the middle stages, and found they could reduce the computational effort by up to 17% with only a small decrease in the quality of the generated text. They used analysis of how much information is lost when using the smaller model at different stages to confirm that the middle steps are the most sensitive.
Why it matters?
This research is important because it offers a practical way to make masked diffusion language models faster and more efficient. By strategically using smaller models during less critical parts of the generation process, we can reduce the computational cost without sacrificing too much quality, making these powerful models more accessible and usable for a wider range of applications.
Abstract
Recent advances in masked diffusion language models (MDLMs) narrow the quality gap to autoregressive LMs, but their sampling remains expensive because generation requires many full-sequence denoising passes with a large Transformer and, unlike autoregressive decoding, cannot benefit from KV caching. In this work, we exploit the flexibility of the diffusion framework and study model scheduling, where a smaller MDLM replaces the full model at a subset of denoising steps. Across models trained on OpenWebText and LM1B, we show that early and late denoising steps are substantially more robust to such replacement than middle steps, enabling up to a 17% reduction in FLOPs with only modest degradation in generative perplexity under both unconditional and prefix-conditional generation, while preserving sample diversity. We support these findings with a step-importance analysis based on loss and KL divergence between small and large models across timesteps, as well as an exhaustive search over coarse step segments, both of which identify the middle of the diffusion trajectory as most sensitive consistently across datasets. Our results suggest that simple, architecture-agnostic scheduling rules can significantly accelerate MDLM sampling while largely preserving generation quality.