CNN-based End-to-end Autonomous Driving on FPGA Using TVM and VTA

Abstract

This paper presents a method reducing inference time and maintaining inference accuracy in autonomous driving using TVM and Versatile Tensor Accelerator (VTA) on Field Programmable Gate Array (FPGA). We focus on End-to-end deep neural networks (DNNs) that directly calculate throttle and steering values of cars using camera images to realize autonomous driving. This network is highly accurate in that it does not add any artificial features. However, real-time implementation of autonomous driving DNNs in embedded systems is problematic due to the limited computational resources and electric power. To address this problem, we implemented the network on an FPGA using TVM and VTA. We modified the network using TVM to (1) reduce the number of bits in the neural network parameters from float32 to int8, (2) schedule the matrix computation in hardware, and (3) optimize the operators, tensors, and hardware parameters to maximize the performance of the neural network at runtime. We measured inference time and accuracy of the CPU and CPU + FPGA implementations on the same board. The experiment shows that CPU+FPGA reduced the inference time by 61%, with a 1 % decrease in inference accuracy than CPU implementation. We conclude that FPGA implementation of the end-to-end autonomous driving network can reduce the inference time and maintain the inference accuracy.