Autonomous Driving with Spiking Neural Networks

🌈 Abstract

The paper presents Spiking Autonomous Driving (SAD), the first end-to-end spiking neural network (SNN) designed for autonomous driving. SAD integrates perception, prediction, and planning into a unified neuromorphic framework, demonstrating competitive performance on the nuScenes dataset while exhibiting exceptional energy efficiency compared to state-of-the-art artificial neural network (ANN) methods.

🙋 Q&A

[01] Spiking Neuron Layer

1. How are spiking neurons represented and what are their key dynamics? Spiking neurons are represented as recurrent neurons with binarized activations and a diagonal recurrent weight matrix. The key dynamics are described by the Leaky Integrate-and-Fire (LIF) model, where the membrane potential integrates the temporal input component and emits a spike if it exceeds the firing threshold.

2. How do spiking neurons enable efficient spatiotemporal processing for autonomous driving tasks? The spiking neuron layer incorporates spatiotemporal information into the hidden state of each neuron (membrane potential), which are then converted into binary spikes emitted to the next layer. This spike-driven computing is well-suited for the dynamic nature of autonomous driving tasks.

[02] Perception Module

1. What are the key components of the perception module? The perception module consists of an encoder and a decoder. The encoder processes multi-view camera inputs to generate features and depth estimations, while the decoder generates BEV segmentation and provides input to the planning module.

2. How does the perception module handle the temporal dimension in the encoder and decoder? The encoder uses sequence repetition (SR) to align the time-varying dimension of the input data with the model, while the decoder uses sequential alignment (SA) to introduce new data instances at each time-step. This combination of strategies leverages the strengths of each approach for effective spatiotemporal processing.

[03] Prediction Module

1. How does the prediction module forecast future states? The prediction module uses a "dual pathway" architecture, where one pathway focuses on encoding information from the past and the other pathway specializes in predicting future information. The embeddings from these two pathways are then fused to integrate past and future information, enabling the anticipation of dynamic changes in the environment.

2. How does the prediction module model future uncertainty? The prediction module models future uncertainty using a conditional Gaussian distribution, where the mean and variance are generated by passing the present feature through separate spiking neural network layers.

[04] Planning Module

1. What are the key components of the planning module? The planning module generates safe trajectories by considering predicted occupancy of space around the vehicle, traffic rules, and ride comfort. It first generates a diverse set of potential trajectories using the bicycle model, and then further refines the "best" trajectory using a Spiking Gated Recurrent Unit (SGRU).

2. How does the SGRU-based optimization enhance the reliability of the planned trajectories? The SGRU-based optimization incorporates features derived from the front camera's encoder, as well as dynamic traffic light information, to mitigate uncertainties in the perceptual and predictive analyses and improve the overall trajectory planning process.

[05] End-to-End Training

1. How is the end-to-end model trained? The end-to-end model is trained using a composite loss function that combines objectives from the perception, prediction, and planning modules. The training process is divided into three stages: (1) training the perception module, (2) training the prediction module, and (3) training the planning module.

2. What are the benefits of the stage-wise training approach? The stage-wise training approach ensures that each component is well-trained before integrating them into the end-to-end framework, which helps to stabilize the loss convergence and improve the overall performance of the model.

[06] Experimental Results

1. How does the SAD model perform compared to state-of-the-art ANN-based methods? The SAD model achieves competitive performance in perception, prediction, and planning tasks on the nuScenes dataset, while offering significant improvements in energy efficiency compared to ANN-based methods.

2. What are the key advantages of the SAD model in terms of energy efficiency? The SAD model's utilization of spiking neurons enables substantial energy reductions, drawing 7.33 times less energy than the Freespace model and 75.03 times less energy than the ST-P3 model during inference.