Block Diffusion: Interpolating Between Autoregressive and Diffusion Language Models
Marianne Arriola, Aaron Gokaslan, Justin T Chiu, Zhihan Yang, Zhixuan Qi, Jiaqi Han, Subham Sekhar Sahoo, Volodymyr Kuleshov
2025-03-13
Summary
This paper talks about a new AI model called Block Diffusion that mixes two text-generation methods to make them faster and able to write any length of text.
What's the problem?
Current AI text generators either write one word at a time (slow but good) or try to write all words at once (fast but limited to fixed lengths and less accurate).
What's the solution?
Block Diffusion splits text into chunks, uses the fast method inside each chunk, and connects chunks like the slow method, while adding smart tricks to keep training stable and efficient.
Why it matters?
This makes AI writing tools quicker and more flexible for tasks like chatbots or story generation, without sacrificing quality or limiting output length.
Abstract
Diffusion language models offer unique benefits over autoregressive models due to their potential for parallelized generation and controllability, yet they lag in likelihood modeling and are limited to fixed-length generation. In this work, we introduce a class of block diffusion language models that interpolate between discrete denoising diffusion and autoregressive models. Block diffusion overcomes key limitations of both approaches by supporting flexible-length generation and improving inference efficiency with KV caching and parallel token sampling. We propose a recipe for building effective block diffusion models that includes an efficient training algorithm, estimators of gradient variance, and data-driven noise schedules to minimize the variance. Block diffusion sets a new state-of-the-art performance among diffusion models on language modeling benchmarks and enables generation of arbitrary-length sequences. We provide the code, along with the model weights and blog post on the project page: https://m-arriola.com/bd3lms/