An active vision retina for virtual reality and telepresence using biologically-motivated neuromorphic CNN

Abstract

We have developed a CNN-based low bandwidth visual telepresence simulation which may be thought of as consisting of two parts, a retina-inspired encoder (attached to a camera at the distant site), and a decoder, a sort of inverse retina, attached to the user's display. The encoder carries out many transformations analogous to those in the actual retina, for instance, discarding information about absolute intensities, unchanging areas of the image, and spatial resolution in the visual scene's periphery. Our goal was to strictly constrain the bandwidth of the channel between encoder and decoder (while freely making use of the potentially enormous computational capacity of CNN) and we demonstrate the feasibility of a useful system using less than 30Kbit/s (telephone-line) bandwidth. A back channel provides the encoder with information about what the user is looking at, making this an active vision system-a first approximation to a true extension of the user's eye.