Beyond Binary Preference: Aligning Diffusion Models to Fine-grained Criteria by Decoupling Attributes
Chenye Meng, Zejian Li, Zhongni Liu, Yize Li, Changle Xie, Kaixin Jia, Ling Yang, Huanghuang Deng, Shiying Ding, Shengyuan Zhang, Jiayi Li, Lingyun Sun
2026-01-09
Summary
This paper focuses on making AI image generators, specifically diffusion models, better at creating images that match what experts consider 'good' quality. Current methods use simple feedback like 'good' or 'bad', which isn't detailed enough to capture nuanced artistic preferences.
What's the problem?
Existing methods for improving AI image generation rely on very simple signals – basically just a score or a thumbs up/down. This is a problem because human judgment of image quality, especially in areas like art, is much more complex. We don't just say an image is 'good'; we can pinpoint *why* it's good or bad, referencing specific artistic qualities. These current methods can't learn from that detailed feedback.
What's the solution?
The researchers created a detailed set of rules, developed with art experts, that breaks down image quality into many specific positive and negative features, organized like a family tree. Then, they trained the AI in two steps. First, they used these rules to teach a separate 'helper' AI what good art looks like. Second, they used a new technique called Complex Preference Optimization to adjust the main AI image generator, making it more likely to create images with the positive features and less likely to create images with the negative ones, based on what the 'helper' AI learned.
Why it matters?
This work is important because it shows how to train AI to understand and create images that align with complex human expertise, going beyond simple 'good' or 'bad' judgments. This could lead to AI art generators that are much more capable of producing high-quality, aesthetically pleasing images that meet specific artistic standards, and opens the door for applying this approach to other areas where detailed, expert feedback is available.
Abstract
Post-training alignment of diffusion models relies on simplified signals, such as scalar rewards or binary preferences. This limits alignment with complex human expertise, which is hierarchical and fine-grained. To address this, we first construct a hierarchical, fine-grained evaluation criteria with domain experts, which decomposes image quality into multiple positive and negative attributes organized in a tree structure. Building on this, we propose a two-stage alignment framework. First, we inject domain knowledge to an auxiliary diffusion model via Supervised Fine-Tuning. Second, we introduce Complex Preference Optimization (CPO) that extends DPO to align the target diffusion to our non-binary, hierarchical criteria. Specifically, we reformulate the alignment problem to simultaneously maximize the probability of positive attributes while minimizing the probability of negative attributes with the auxiliary diffusion. We instantiate our approach in the domain of painting generation and conduct CPO training with an annotated dataset of painting with fine-grained attributes based on our criteria. Extensive experiments demonstrate that CPO significantly enhances generation quality and alignment with expertise, opening new avenues for fine-grained criteria alignment.