Temporally-consistent 3D Reconstruction of Birds
Johannes Hägerlind, Jonas Hentati-Sundberg, Bastian Wandt
2024-08-28
Summary
This paper discusses a method for creating 3D models of seabirds, specifically the common murre, to help scientists study their behavior and physical characteristics over time.
What's the problem?
Studying seabirds is important for understanding environmental changes, but creating accurate 3D models of these birds is challenging. Birds move quickly and in unpredictable ways, making it hard to capture their shape and motion accurately using traditional methods.
What's the solution?
The authors propose a new approach that uses videos to reconstruct the 3D shape and pose of the common murre. Their method includes several steps: detecting the birds in the video, tracking their movements, segmenting them from the background, and then creating a consistent 3D model over time. They also introduce a new technique called temporal loss, which helps improve the accuracy of the model by considering how the birds move across different frames in the video. Additionally, they provide a dataset with 10,000 video frames that show multiple birds in various poses and interactions.
Why it matters?
This research is significant because it allows scientists to analyze seabird behavior and physical changes more effectively. By improving how we can create detailed 3D models of animals in motion, this method can help in conservation efforts and studies related to environmental changes.
Abstract
This paper deals with 3D reconstruction of seabirds which recently came into focus of environmental scientists as valuable bio-indicators for environmental change. Such 3D information is beneficial for analyzing the bird's behavior and physiological shape, for example by tracking motion, shape, and appearance changes. From a computer vision perspective birds are especially challenging due to their rapid and oftentimes non-rigid motions. We propose an approach to reconstruct the 3D pose and shape from monocular videos of a specific breed of seabird - the common murre. Our approach comprises a full pipeline of detection, tracking, segmentation, and temporally consistent 3D reconstruction. Additionally, we propose a temporal loss that extends current single-image 3D bird pose estimators to the temporal domain. Moreover, we provide a real-world dataset of 10000 frames of video observations on average capture nine birds simultaneously, comprising a large variety of motions and interactions, including a smaller test set with bird-specific keypoint labels. Using our temporal optimization, we achieve state-of-the-art performance for the challenging sequences in our dataset.