ST-FlowNet: An efficient Spiking Neural Network for event-based optical flow estimation

Abstract

Spiking Neural Networks (SNNs) have emerged as a promising tool for event-based optical flow estimation tasks due to their capability for spatio-temporal information processing and low-power computation. However, the performance of SNN models is often constrained, limiting their applications in real-world scenarios. To address this challenge, we propose ST-FlowNet, a novel neural network architecture specifically designed for optical flow estimation from event-based data. The ST-FlowNet architecture integrates ConvGRU modules to facilitate cross-modal feature augmentation and temporal alignment of the predicted optical flow, thereby improving the network’s ability to capture complex motion patterns. Additionally, we introduce two strategies for deriving SNN models from pre-trained artificial neural networks (ANNs): a standard ANN-to-SNN conversion pipeline and our proposed BISNN method. Notably, the BISNN method alleviates the complexities involved in selecting biologically inspired parameters, further enhancing the robustness of SNNs for optical flow estimation tasks. Extensive evaluations on three benchmark event-based datasets demonstrate that the SNN-based ST-FlowNet model outperforms state-of-the-art methods, achieving superior accuracy in optical flow estimation across a diverse range of dynamic visual scenes. Furthermore, the energy efficiency of models also underscores the potential of SNNs for practical deployment in energy-constrained environments. Overall, our work presents a novel framework for optical flow estimation using SNNs and event-based data, contributing to the advancement of neuromorphic vision applications.