Hardware-friendly fast rate-distortion optimization algorithm for AV1 encoder

The rate distortion optimization (RDO) process aims at achieving optimal coding performance by determining the optimal coding mode according to a certain strategy in the AV1 video coding. However, the high computational complexity and strong data dependency in RDO impede real-time applications. To address these issues, a fast RDO algorithm suitable for hardware implementation is proposed. Firstly, we propose a high-frequency coefficients zero-setting approach to optimize the hardware memory occupation. Then, in the rate-distortion calculation stage, an efficient rate estimation method is proposed based on a statistical feature for the number of quantization coefficients, and the distortion estimation method is proposed by considering intrinsic features in the all-zero block. Finally, a reconstruction approximate model is proposed to solve the low parallelism issue caused by the coupling of pixel reconstruction and prediction data. Experimental results show that the proposed algorithm achieves 68.49% and 50.77% time-saving by 2.73% and 2.95% Bjøntegaard delta rate (BD-Rate) increase on average under all intra (AI) and random access (RA) configurations, respectively.