MedSAMix: A Training-Free Model Merging Approach for Medical Image Segmentation
Yanwu Yang, Guinan Su, Jiesi Hu, Francesco Sammarco, Jonas Geiping, Thomas Wolfers
2025-08-20
Summary
This paper introduces a new method called MedSAMix that combines the strengths of general medical image models and specialized ones to make them better at segmenting different parts of medical images.
What's the problem?
While general vision models like SAM are good, when they are specifically trained for medical images (like MedSAM), they don't always perform well on all types of medical scans because the medical data they're trained on can be inconsistent, have limited labels, or be different from what they're used to. This makes it hard for them to be useful for every medical segmentation job.
What's the solution?
The researchers developed MedSAMix, a way to merge different models without needing to retrain them from scratch. It uses a smart, automated process to figure out the best way to combine models, specifically by optimizing how layers from a general model and a specialized medical model are blended together. They also created two ways to use this for clinical settings: one for highly specific tasks and another for broader, multi-task applications.
Why it matters?
MedSAMix improves how well medical image segmentation models work, both for very specific medical conditions and for a wider range of tasks. This means doctors can get more accurate results when analyzing medical scans, leading to better diagnoses and treatments across many different medical imaging scenarios.
Abstract
Universal medical image segmentation models have emerged as a promising paradigm due to their strong generalizability across diverse tasks, showing great potential for a wide range of clinical applications. This potential has been partly driven by the success of general-purpose vision models such as the Segment Anything Model (SAM), which has inspired the development of various fine-tuned variants for medical segmentation tasks. However, fine-tuned variants like MedSAM are trained on comparatively limited medical imaging data that often suffers from heterogeneity, scarce annotations, and distributional shifts. These challenges limit their ability to generalize across a wide range of medical segmentation tasks. In this regard, we propose MedSAMix, a training-free model merging method that integrates the strengths of both generalist models (e.g., SAM) and specialist models (e.g., MedSAM) for medical image segmentation. In contrast to traditional model merging approaches that rely on manual configuration and often result in suboptimal outcomes, we propose a zero-order optimization method to automatically discover optimal layer-wise merging solutions. Furthermore, for clinical applications, we develop two regimes to meet the demand of domain-specificity and generalizability in different scenarios by single-task optimization and multi-objective optimization respectively. Extensive evaluations on 25 medical segmentation tasks demonstrate that MedSAMix effectively mitigates model bias and consistently improves performance in both domain-specific accuracy and generalization, achieving improvements of 6.67% on specialized tasks and 4.37% on multi-task evaluations.