RSCaMa: Remote Sensing Image Change Captioning with State Space Model

🌈 Abstract

The paper introduces a novel approach called RSCaMa (Remote Sensing Change Captioning Mamba) for the task of Remote Sensing Image Change Captioning (RSICC). The key contributions are:

• Introducing State Space Models (SSMs), particularly Mamba, into the RSICC task, providing a benchmark for Mamba-based RSICC.
• Proposing the CaMa layer consisting of Spatial Difference-guided SSM (SD-SSM) and Temporal Traveling SSM (TT-SSM) to enhance spatial change perception and temporal interaction between bi-temporal features.
• Demonstrating the superior performance of RSCaMa compared to state-of-the-art methods, and highlighting the potential of Mamba in the RSICC task.
• Systematically evaluating different language decoding schemes, including Mamba, GPT-style, and Transformer decoders, to provide valuable insights for future RSICC research.

🙋 Q&A

[01] Introduction

1. What is the main task addressed in this paper? The main task addressed in this paper is Remote Sensing Image Change Captioning (RSICC), which aims to identify surface changes in multi-temporal remote sensing images and describe them in natural language.

2. What are the key components of the current mainstream RSICC methods? Current mainstream RSICC methods typically employ an encoder-decoder structure. The encoder uses a CNN or Vision Transformer (ViT) as the backbone to extract visual features from bi-temporal images, and a well-designed "neck" module to strengthen the spatial-temporal correlation of bi-temporal features and capture interesting change features. The decoder then converts these visual features into captions using language models such as recurrent neural networks or Transformers.

3. What is the motivation for introducing State Space Models (SSMs), particularly Mamba, into the RSICC task? Recently, State Space Models (SSMs), especially Mamba, have demonstrated outstanding performance in many fields due to their efficient feature-selective modelling capability. However, their potential in the RSICC task remains unexplored. The paper aims to introduce Mamba into RSICC and propose a novel approach called RSCaMa to leverage the advantages of Mamba for efficient spatial-temporal modelling in this multi-modal task.

[02] Methodology

1. What are the main components of the proposed RSCaMa architecture? The RSCaMa architecture consists of three main components:

• Backbone: Siamese backbones to extract bi-temporal features
• CaMa layers: Multiple layers consisting of Spatial Difference-guided SSM (SD-SSM) and Temporal Traveling SSM (TT-SSM) to enhance spatial change perception and temporal interaction
• Language decoder: Transformer decoder with cross-attention mechanism to generate descriptive captions

2. How does the SD-SSM module work? The SD-SSM module performs bidirectional scanning on the flattened visual tokens to achieve spatial-aware understanding. It also multiplies the bi-temporal differencing features and the output of the bidirectional SSMs to guide the model and enhance its perception of changes.

3. What is the purpose of the TT-SSM module? The TT-SSM module aims to perform temporal interaction between the bi-temporal features. It rearranges the bi-temporal token sequences in an interleaving manner to facilitate effective temporal information exchange, and then processes the rearranged sequence using bidirectional SSM.

[03] Experiments

1. What dataset was used for the experiments? The experiments were conducted on the LEVIR-CC dataset, a large-scale RSICC dataset comprising 10,077 pairs of remote sensing images capturing changes over time in 20 distinct regions across Texas, USA.

2. How does the performance of RSCaMa compare to the state-of-the-art methods? The experimental results show that RSCaMa demonstrates outstanding performance, particularly excelling in key metrics such as BLEU-4 and CIDEr. Compared to the latest PromptCC method, RSCaMa achieves a +1.70% improvement on BLEU-4 and a +1.11% improvement on CIDEr. This significant enhancement is attributed to the innovative spatial-temporal modelling units (SD-SSM and TT-SSM) in the CaMa layers.

3. What insights are provided by the ablation studies on the CaMa layers and language decoders? The ablation studies validate the effectiveness of the SD-SSM and TT-SSM modules in the CaMa layers, demonstrating their contributions to improving change captioning performance. The comparison of different language decoders (Mamba, GPT-style, and Transformer) suggests that the Transformer decoder with cross-attention mechanism performs best in transforming visual to textual information, proving the superiority of cross-attention in cross-modal understanding.