KeySync: A Robust Approach for Leakage-free Lip Synchronization in High Resolution
Antoni Bigata, Rodrigo Mira, Stella Bounareli, Michał Stypułkowski, Konstantinos Vougioukas, Stavros Petridis, Maja Pantic
2025-05-02
Summary
This paper talks about KeySync, a new system that makes sure computer-generated talking faces have perfectly matched lip movements, even in high-quality videos.
What's the problem?
When creating videos where a character's lips move to match speech, it's common for mistakes to happen, like the lips showing the wrong expressions or getting blocked by things like hands or objects, which makes the video look fake.
What's the solution?
The researchers built a two-step process that first focuses on getting the lip movements right without letting extra facial expressions leak in, and then fixes any problems caused by things blocking the lips, resulting in much more realistic and accurate talking faces.
Why it matters?
This matters because it helps make digital characters and video edits look much more believable, which is really important for movies, video games, virtual meetings, and any place where realistic talking faces are needed.
Abstract
KeySync, a two-stage framework, addresses expression leakage and occlusions in lip synchronization, improving visual quality and achieving state-of-the-art results in lip reconstruction and cross-synchronization.