Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticket

🌈 Abstract

The paper proposes Spikformer, a Spiking Neural Network (SNN) that integrates the self-attention mechanism and Transformer architecture, to improve the performance of SNNs. To further enhance the Spikformer, the authors introduce Spikformer V2, which includes a Spiking Convolutional Stem (SCS) module and leverages self-supervised learning. The key contributions are:

1. Spiking Self-Attention (SSA) mechanism that enables SNNs to capture interdependencies without the need for softmax normalization.
2. Spikformer V2 with SCS to mitigate information loss and improve performance on large-scale datasets like ImageNet.
3. Exploration of self-supervised learning for SNNs, specifically masked image modeling, to train larger and deeper Spikformer V2 models.
4. Achieving state-of-the-art SNN performance on ImageNet, surpassing 80% accuracy for the first time.

🙋 Q&A

[01] Spikformer

1. What are the key components of the Spikformer architecture? The Spikformer architecture consists of the following key components:

• Spiking Patch Splitting (SPS) module to convert the input image into a sequence of spike-form patches
• Spiking Self-Attention (SSA) mechanism in the Spikformer encoder to model interdependencies in the spike-form features
• Multi-Layer Perceptron (MLP) block in the Spikformer encoder
• Global Average-Pooling and Classification Head to output the prediction

2. How does the Spiking Self-Attention (SSA) mechanism differ from the vanilla self-attention used in ANNs? The key differences are:

• SSA operates on spike-form Query, Key, and Value, whereas vanilla self-attention uses float-point forms.
• SSA eliminates the softmax normalization step, as the spike-form inputs are inherently non-negative.
• SSA uses logical AND and addition operations instead of float-point matrix multiplications, making it more efficient for SNNs.

3. What are the advantages of the SSA mechanism over the vanilla self-attention in SNNs? The advantages of SSA are:

• It is more compatible with the computational characteristics of SNNs, as it avoids float-point operations and softmax.
• It has reduced computational complexity and energy consumption compared to vanilla self-attention.
• It is better suited for handling spike sequences with restricted information content compared to vanilla self-attention.

[02] Spikformer V2

1. What are the key differences between Spikformer and Spikformer V2? The main differences are:

• Spikformer V2 replaces the Spiking Patch Splitting (SPS) module with a Spiking Convolutional Stem (SCS) module to mitigate information loss and improve performance on large-scale datasets.
• Spikformer V2 explores self-supervised learning, specifically masked image modeling, to train larger and deeper models.

2. How does the Spiking Convolutional Stem (SCS) module differ from the original SPS module? The key differences are:

• SCS uses standard 2D convolution layers for downsampling instead of max-pooling, to avoid information loss.
• SCS has an increased number of convolution layers, with two consecutive layers forming an MLP-like structure to enhance feature learning.

3. How does the self-supervised learning approach benefit the performance of Spikformer V2? The self-supervised learning approach, specifically the masked image modeling, helps in the following ways:

• It enables training of larger and deeper Spikformer V2 models, which further unleashes the potential of SSA and improves the overall performance.
• It leads to stable training and better performance compared to direct supervised training, especially for larger model sizes.
• It reduces the training cost compared to direct supervised training, with the acceleration ratio becoming more prominent as the model size increases.