SAM2-UNet: Segment Anything 2 Makes Strong Encoder for Natural and Medical Image Segmentation

🌈 Abstract

The paper proposes SAM2-UNet, a simple yet effective U-shaped framework for versatile image segmentation tasks. The key points are:

• Simplicity: SAM2-UNet adopts a classic U-shaped encoder-decoder architecture.
• Efficiency: Adapters are integrated into the encoder to enable parameter-efficient fine-tuning.
• Effectiveness: Extensive experiments on 18 public datasets across 5 challenging benchmarks demonstrate that SAM2-UNet delivers powerful performance.

🙋 Q&A

[01] Introduction

1. What is the motivation behind the proposed SAM2-UNet framework?

• Image segmentation is a crucial task in computer vision, serving as the foundation for various applications.
• The emergence of vision foundation models (VFMs) like SAM1 and SAM2 has introduced significant potential in image segmentation.
• However, SAM2 still produces class-agnostic segmentation results when no manual prompt is provided, limiting its adaptability to downstream tasks.
• The paper aims to explore strategies to enhance SAM2's adaptability and performance in downstream segmentation tasks.

2. What are the key components of the SAM2-UNet architecture?

• Encoder: Adopts the Hiera backbone pretrained by SAM2, which has a hierarchical structure suitable for U-shaped networks.
• Decoder: Follows the classic U-Net design, consisting of three decoder blocks.
• Receptive Field Blocks (RFBs): Used to reduce the channel number and enhance the lightweight features.
• Adapters: Inserted before each multi-scale block of the Hiera encoder to enable parameter-efficient fine-tuning.

[02] Experiments

1. What are the key benchmarks and datasets used in the experiments?

• The experiments are conducted on 5 different benchmarks with 18 datasets in total:
  • Camouflaged Object Detection (4 datasets)
  • Salient Object Detection (5 datasets)
  • Marine Animal Segmentation (2 datasets)
  • Mirror Detection (2 datasets)
  • Polyp Segmentation (5 datasets)

2. How does SAM2-UNet perform compared to state-of-the-art methods across the different benchmarks?

• SAM2-UNet consistently outperforms existing specialized state-of-the-art methods across all 5 benchmarks, achieving the highest scores in every metric.
• For example, in Camouflaged Object Detection, SAM2-UNet surpasses the previous best method FEDER by 1.1-4.8% in S-measure.
• In Salient Object Detection, SAM2-UNet outperforms MENet by 1.4-3.4% in S-measure.
• For Marine Animal Segmentation, SAM2-UNet outperforms the second-best MASNet by 0.7-5.7% in mIoU.
• In Mirror Detection, SAM2-UNet significantly outperforms HetNet by 3.8-9% in IoU.
• In Polyp Segmentation, SAM2-UNet delivers state-of-the-art performance on 3 out of 5 datasets, exceeding CFA-Net by 1.3-6.5% in mDice.

3. What is the impact of the Hiera backbone size on the performance of SAM2-UNet?

• The ablation study shows that a larger Hiera backbone typically results in better performance.
• Even with the smaller Hiera-Base+ backbone, SAM2-UNet still surpasses FEDER and delivers satisfactory results.
• As the backbone size decreases further, SAM2-UNet also produces results comparable to PFNet and ZoomNet, demonstrating the high-quality representations provided by the SAM2 pre-trained Hiera backbone.

[03] Conclusion

1. What are the key contributions of the proposed SAM2-UNet framework?

• SAM2-UNet is a simple yet effective U-shaped framework for versatile segmentation across natural and medical domains.
• It features a SAM2 pre-trained Hiera encoder coupled with a classic U-Net decoder, designed for ease of understanding and use.
• Extensive experiments demonstrate the effectiveness of SAM2-UNet, which can serve as a new baseline for developing future SAM2 variants.