Spiking Depth: Depth estimation from sparse events with spiking neural networks

Abstract

Event cameras provide remarkable temporal resolution, wide dynamic range, and low power consumption, making them ideal for depth estimation in high-contrast and dynamic environments. While spiking neural networks (SNNs) are naturally suited to process event data, their performance in depth estimation tasks has not consistently surpassed those of traditional artificial neural networks (ANNs) because of the former’s lack of effective mechanisms for handling the sparse nature of event data. Herein, we propose Spiking Depth, a novel end-to-end SNN framework designed to overcome the limitations of current ANN models and achieve superior depth estimation from sparse event data. In particular, Spiking Depth introduces three key innovations: an event encoding module based on a spiking-driven fusion block (SDFB), enhanced skip connections incorporating both SDFB and an adaptive spiking convolutional block attention module, and the event depth loss that optimizes depth estimation by addressing the sparse and dynamic nature of event data. Spiking Depth outperforms current state-of-the-art SNN and ANN models on two event-based datasets: the Multi Vehicle Stereo Event Camera (MVSEC) dataset, which is a real-world dataset, and a synthetic dataset. On the MVSEC dataset, our model achieves mean depth error values of 11.8 cm, 18.0 cm, and 12.5 cm for Splits 1, 2, and 3, respectively, setting a new benchmark for event-based depth estimation with significantly lower power consumption.