Reconfigurable in-sensor processing based on a multi-phototransistor–one-memristor array

Memristors with photonic sensory capabilities can be used as elements in machine vision systems but face challenges in terms of encoding and processing optical data. This has led to different neural network architectures being developed for specific vision tasks, which limits progress towards more versatile in-sensor vision computing platforms. Here we describe a multi-phototransistor and one-memristor array that is based on niobium oxide memristors. It has reconfigurable dynamics and is compatible with both machine learning (analogue) and bioinspired (spiking) neural network architectures. The array can sense and process optical images and synchronize spatio-temporal data across different encoding formats. When the array is coupled with a classifier network using a one-transistor and one-memristor non-volatile memory array, it supports a variety of optical neural networks (including optical convolutional neural networks, recurrent neural networks and spiking neural networks). The resulting system can perform various computing vision tasks, such as recognizing static, motion and colour images. Niobium oxide memristors with reconfigurable dynamics can be used to make an array integrated with phototransistors that can encode image information in analogue or spiking form and can support different neural network architectures for image processing tasks.