Inference-Time Scaling for Diffusion Models beyond Scaling Denoising Steps
Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, Saining Xie
2025-01-17
Summary
This paper talks about a new way to make AI models that create images (called diffusion models) work better without changing how they're trained. Instead, it focuses on improving how these models work when they're actually making images.
What's the problem?
Current diffusion models can be adjusted to use more steps when creating images, which can improve quality. But this improvement stops after a certain point, usually after a few dozen steps. This means there's a limit to how much better we can make the images just by adding more steps.
What's the solution?
The researchers came up with a new approach that goes beyond just adding more steps. They created a system that searches for better 'noise' (the random starting point for image creation) during the image-making process. They use two main tools: 'verifiers' that check if an image is good, and algorithms that look for better noise to start with. They tested this on different types of image creation tasks and found that using more computing power in this way can make the images much better.
Why it matters?
This matters because it shows we can make AI-generated images much better without having to retrain the entire model, which is time-consuming and expensive. It's like finding a way to make a car go faster without rebuilding the engine, just by changing how you drive it. This could lead to better AI-generated images for things like art, design, or even helping to create visual content for movies or games, all without needing to create entirely new AI models.
Abstract
Generative models have made significant impacts across various domains, largely due to their ability to scale during training by increasing data, computational resources, and model size, a phenomenon characterized by the scaling laws. Recent research has begun to explore inference-time scaling behavior in Large Language Models (LLMs), revealing how performance can further improve with additional computation during inference. Unlike LLMs, diffusion models inherently possess the flexibility to adjust inference-time computation via the number of denoising steps, although the performance gains typically flatten after a few dozen. In this work, we explore the inference-time scaling behavior of diffusion models beyond increasing denoising steps and investigate how the generation performance can further improve with increased computation. Specifically, we consider a search problem aimed at identifying better noises for the diffusion sampling process. We structure the design space along two axes: the verifiers used to provide feedback, and the algorithms used to find better noise candidates. Through extensive experiments on class-conditioned and text-conditioned image generation benchmarks, our findings reveal that increasing inference-time compute leads to substantial improvements in the quality of samples generated by diffusion models, and with the complicated nature of images, combinations of the components in the framework can be specifically chosen to conform with different application scenario.