Exploring Joint Embedding Architectures and Data Augmentations for Self-Supervised Representation Learning in Event-Based Vision

Abstract

This paper proposes a self-supervised representation learning (SSRL) framework for event-based vision, which leverages various lightweight convolutional neural networks (CNNs) including 2D-, 3D-, and Spiking CNNs. The method uses a joint embedding architecture to maximize the agreement between features extracted from different views of the same event sequence. Popular event data augmentation techniques are employed to design an efficient augmentation policy for event-based SSRL, and we provide novel data augmentation methods to enhance the pretraining pipeline. Given the novelty of SSRL for event-based vision, we elaborate standard evaluation protocols and use them to evaluate our approach. Our study demonstrates that pretrained CNNs acquire effective and transferable features, enabling them to achieve competitive performance in object or action recognition across various commonly used event-based datasets, even in a low-data regime. This paper also conducts an experimental analysis of the extracted features regarding the Uniformity-Tolerance tradeoff to assess their quality, and measure the similarity of representations using linear Center Kernel Alignement. These quantitative measurements reinforce our observations from the performance benchmarks and show substantial differences between the learned representations of all types of CNNs despite being optimized with the same approach.