Optimization Method and Implementation of Fake Quantization from the Perspective of Hardware Performance

Abstract

Deep learning networks can be accelerated by reducing the overall network volume using quantization or pruning techniques. The well-known quantization technique is Post Training Quantization (PTQ) and Quantization Aware Training (QAT). We applied an INT8 quantized network to design deep learning acceleration hardware and found that the performance of the deep learning network deteriorated due to errors occurring in the mult/shift based re-quantization step. This quantization error becomes a bigger problem in the training process rather than inference, and the FP32 arithmetic operator is applied to prevent the resulting accuracy drop. In this paper, we investigate whether the use of FP32 operators can outperform employing mult/shift operators under specific conditions. We accomplish this by analyzing the data flow based on output channel tiling and conducting a size analysis of the implemented hardware.