SPF-Portrait: Towards Pure Portrait Customization with Semantic Pollution-Free Fine-tuning
Xiaole Xian, Zhichao Liao, Qingyu Li, Wenyu Qin, Pengfei Wan, Weicheng Xie, Long Zeng, Linlin Shen, Pingfa Feng
2025-04-07
Summary
This paper talks about SPF-Portrait, a smart AI tool that customizes portrait photos using text descriptions without messing up other details like facial features or backgrounds.
What's the problem?
Current AI portrait editors accidentally change unwanted parts of photos (like making someone's face look different) when trying to modify specific features like hairstyles or expressions.
What's the solution?
SPF-Portrait uses two AI models working together - one keeps the original photo's style intact, while the other makes changes only where needed, guided by special maps that highlight exact areas to modify.
Why it matters?
This helps people create personalized portraits (like avatars or profile pictures) that keep their original look while adding new features, making AI photo editing more reliable for social media or professional use.
Abstract
Fine-tuning a pre-trained Text-to-Image (T2I) model on a tailored portrait dataset is the mainstream method for text-driven customization of portrait attributes. Due to Semantic Pollution during fine-tuning, existing methods struggle to maintain the original model's behavior and achieve incremental learning while customizing target attributes. To address this issue, we propose SPF-Portrait, a pioneering work to purely understand customized semantics while eliminating semantic pollution in text-driven portrait customization. In our SPF-Portrait, we propose a dual-path pipeline that introduces the original model as a reference for the conventional fine-tuning path. Through contrastive learning, we ensure adaptation to target attributes and purposefully align other unrelated attributes with the original portrait. We introduce a novel Semantic-Aware Fine Control Map, which represents the precise response regions of the target semantics, to spatially guide the alignment process between the contrastive paths. This alignment process not only effectively preserves the performance of the original model but also avoids over-alignment. Furthermore, we propose a novel response enhancement mechanism to reinforce the performance of target attributes, while mitigating representation discrepancy inherent in direct cross-modal supervision. Extensive experiments demonstrate that SPF-Portrait achieves state-of-the-art performance. Project webpage: https://spf-portrait.github.io/SPF-Portrait/