Inconsistencies In Consistency Models: Better ODE Solving Does Not Imply Better Samples
Noël Vouitsis, Rasa Hosseinzadeh, Brendan Leigh Ross, Valentin Villecroze, Satya Krishna Gorti, Jesse C. Cresswell, Gabriel Loaiza-Ganem
2024-11-15
Summary
This paper discusses the challenges and findings related to consistency models in generating high-quality samples from diffusion models, revealing that better solving of ordinary differential equations (ODEs) does not necessarily lead to better sample quality.
What's the problem?
Diffusion models are powerful tools for generating high-quality images or data, but they require a lot of computational resources and time due to their iterative sampling process. Consistency models (CMs) were developed to make this process faster and cheaper, but it was unclear how well they actually perform in terms of generating quality samples compared to traditional ODE solvers.
What's the solution?
The authors introduced a new approach called Direct CMs, which directly minimizes the error when solving the ODE associated with diffusion models. They found that while Direct CMs reduced the error in solving the ODE compared to traditional CMs, this improvement led to worse quality in the generated samples. This unexpected result raised questions about why consistency models work well despite their limitations.
Why it matters?
This research is important because it challenges the assumption that improving the mathematical accuracy of ODE solutions will automatically enhance the quality of generated samples. Understanding these inconsistencies can help researchers develop better methods for generating high-quality data and improve the overall performance of diffusion models.
Abstract
Although diffusion models can generate remarkably high-quality samples, they are intrinsically bottlenecked by their expensive iterative sampling procedure. Consistency models (CMs) have recently emerged as a promising diffusion model distillation method, reducing the cost of sampling by generating high-fidelity samples in just a few iterations. Consistency model distillation aims to solve the probability flow ordinary differential equation (ODE) defined by an existing diffusion model. CMs are not directly trained to minimize error against an ODE solver, rather they use a more computationally tractable objective. As a way to study how effectively CMs solve the probability flow ODE, and the effect that any induced error has on the quality of generated samples, we introduce Direct CMs, which directly minimize this error. Intriguingly, we find that Direct CMs reduce the ODE solving error compared to CMs but also result in significantly worse sample quality, calling into question why exactly CMs work well in the first place. Full code is available at: https://github.com/layer6ai-labs/direct-cms.