Neuromorphic visuotactile slip perception for robotic manipulation

Visuotactile sensing technology has received extensive attention in the tactile sensing community due to its stable high-resolution deformation sensing capabilities. However, the existing visuotactile sensing methods are far from humanoid neural information processing mechanism. To address this gap, we propose a neuromorphic visuotactile slip detection method named VT-SNN using Tactile Address-Event Representation (TAER) encoding combined with brain-inspired Spiking Neural Network (SNN) modeling in this paper. Our extensive experimental results demonstrate that the VT-SNN achieves slip detection accuracy of 99.59% and F1 score of 99.28%, which is comparable to Artificial Neural Networks (ANNs) while exhibiting significant advantages in power dissipation and inference time. Furthermore, we deployed the VT-SNN on Intel neuromorphic computing chip–Loihi and performed closed-loop slip-feedback robotic manipulation tasks such as bottle-cap tightening and loosening. Our closed-loop neuromorphic visuotactile sensing system shows significant promise for high accuracy, low latency, and low power dissipation for robotic dexterous manipulation.