Enhanced hierarchical intra mode decision for H.264/AVC

Fast intra mode decision algorithms in H.264/AVC reduce the encoder computational complexity significantly by eliminating some unlikely prediction modes. However, the encoder speedup typically comes with the sacrifice of the quality loss while such degradation is not negligible in many scenarios. To maintain the encoder speedup enabled by the fast intra mode decision without noticeable quality degradation, a method that estimates the reliability of these fast algorithms based on the rate-distortion (R-D) cost ranking is proposed in this paper. By adding few most probable modes conditionally, the proposed method could improve the video quality greatly compared with the most recent hierarchical intra mode decision (HIMD) scheme in the H.264/AVC, where the B-jontegaard Delta Rate (BD-Rate) loss of the enhanced HIMD decreases from 3.04% to 1.16% on average with just 4.47% increase in encoding time, compared with the default HIMD, both against the full rate-distortion optimization solution.