Birch: A Real-Time Multi-Domain Multi-Task Extended Reality Perception Accelerator

Abstract

Birch is a system-on-chip (SoC) that efficiently and accurately accelerates the multi-task multi-domain extended reality (XR) perception pipeline, with workloads such as visual inertial odometry (VIO), eye gaze tracking, and scene understanding. Birch features vision modules with cascaded line buffers, in-step feature sorting, and double-buffered optical flow to extract and track quality features for VIO, a configurable matrix unit and vector unit for acceleration of deep neural network (DNN)-based tasks like eye gaze tracking and scene understanding, and reuse of the DNN vector unit for acceleration of matrix multiplications in VIO to save area in small-form-factor XR devices. Birch extracts and tracks features in 7.14 ms and 2.29 mJ/frame. For DNNs, Birch runs ResNet-18 at 109 frames/s and eye gaze extraction at 985 frames/s, $3.5\times $ and $1.6\times $ faster than prior work, respectively. ResNet-50 also runs in real time at 46.5 frames/s.