Surgical SAM 2: Real-time Segment Anything in Surgical Video by Efficient Frame Pruning
Haofeng Liu, Erli Zhang, Junde Wu, Mingxuan Hong, Yueming Jin
2024-08-19
Summary
This paper introduces Surgical SAM 2 (SurgSAM-2), an advanced model that improves the process of segmenting surgical videos in real-time by efficiently managing which frames to keep for analysis.
What's the problem?
Segmenting surgical videos is important for improving surgery quality and patient outcomes, but existing models like Segment Anything Model 2 (SAM2) are slow and require a lot of computing power to process high-resolution images. This makes it difficult to use them effectively during surgeries.
What's the solution?
SurgSAM-2 uses a technique called Efficient Frame Pruning (EFP) to only keep the most important frames from the video. This reduces the amount of memory needed and speeds up processing while still maintaining high accuracy in segmenting the videos. The model can process frames three times faster than SAM2 and works well even when using lower-resolution data.
Why it matters?
This research is significant because it allows for real-time analysis of surgical videos, making it easier for surgeons to get immediate feedback during operations. By improving the efficiency of video segmentation, it can enhance surgical procedures and ultimately lead to better patient care.
Abstract
Surgical video segmentation is a critical task in computer-assisted surgery and is vital for enhancing surgical quality and patient outcomes. Recently, the Segment Anything Model 2 (SAM2) framework has shown superior advancements in image and video segmentation. However, SAM2 struggles with efficiency due to the high computational demands of processing high-resolution images and complex and long-range temporal dynamics in surgical videos. To address these challenges, we introduce Surgical SAM 2 (SurgSAM-2), an advanced model to utilize SAM2 with an Efficient Frame Pruning (EFP) mechanism, to facilitate real-time surgical video segmentation. The EFP mechanism dynamically manages the memory bank by selectively retaining only the most informative frames, reducing memory usage and computational cost while maintaining high segmentation accuracy. Our extensive experiments demonstrate that SurgSAM-2 significantly improves both efficiency and segmentation accuracy compared to the vanilla SAM2. Remarkably, SurgSAM-2 achieves a 3times FPS compared with SAM2, while also delivering state-of-the-art performance after fine-tuning with lower-resolution data. These advancements establish SurgSAM-2 as a leading model for surgical video analysis, making real-time surgical video segmentation in resource-constrained environments a feasible reality.