2D Gaussian Splatting with Semantic Alignment for Image Inpainting
Hongyu Li, Chaofeng Chen, Xiaoming Li, Guangming Lu
2025-09-12
Summary
This paper explores using a new 3D representation technique, called Gaussian Splatting, for fixing missing parts of 2D images – a process called image inpainting. It shows how this method, originally used for 3D modeling and improving image resolution, can be adapted to realistically fill in damaged or incomplete pictures.
What's the problem?
Image inpainting is tricky because you need to not only make the filled-in area look good up close (locally coherent) but also make sure it fits the overall scene and makes sense (globally consistent). Existing methods often struggle with both at the same time, especially when large portions of the image are missing. Simply filling in pixels can lead to blurry or unrealistic results that don't match the surrounding context.
What's the solution?
The researchers created a system that represents an image as a collection of 2D Gaussians, which are like blurry blobs of color. They then 'splat' these Gaussians onto the image to reconstruct it. Because this method works with a continuous representation, it naturally creates smooth, realistic results. To make it faster and use less memory, they process the image in smaller patches. To ensure the filled-in areas make sense with the rest of the image, they use information from a pre-trained image understanding model (DINO) to guide the process, making sure the new content aligns with the existing scene's overall meaning.
Why it matters?
This work is important because it opens up a new way to approach image inpainting by leveraging the power of Gaussian Splatting. It demonstrates that techniques originally developed for 3D graphics can be successfully applied to 2D image editing, achieving results that are competitive with existing methods and potentially leading to even more advanced image manipulation tools in the future.
Abstract
Gaussian Splatting (GS), a recent technique for converting discrete points into continuous spatial representations, has shown promising results in 3D scene modeling and 2D image super-resolution. In this paper, we explore its untapped potential for image inpainting, which demands both locally coherent pixel synthesis and globally consistent semantic restoration. We propose the first image inpainting framework based on 2D Gaussian Splatting, which encodes incomplete images into a continuous field of 2D Gaussian splat coefficients and reconstructs the final image via a differentiable rasterization process. The continuous rendering paradigm of GS inherently promotes pixel-level coherence in the inpainted results. To improve efficiency and scalability, we introduce a patch-wise rasterization strategy that reduces memory overhead and accelerates inference. For global semantic consistency, we incorporate features from a pretrained DINO model. We observe that DINO's global features are naturally robust to small missing regions and can be effectively adapted to guide semantic alignment in large-mask scenarios, ensuring that the inpainted content remains contextually consistent with the surrounding scene. Extensive experiments on standard benchmarks demonstrate that our method achieves competitive performance in both quantitative metrics and perceptual quality, establishing a new direction for applying Gaussian Splatting to 2D image processing.