Deep Inter Coding with Interpolated Reference Frame for Hierarchical Coding Structure

Abstract

In the hybrid video coding framework, inter prediction is an efficient tool to exploit temporal redundancy. Since the performance of inter prediction depends on the content of reference frames, coding efficiency can be significantly improved by having more effective reference frames. In this paper, we propose an enhanced inter coding scheme by generating artificial reference frames with deep neural network. Specifically, a new reference frame is interpolated from two-sided previously reconstructed frames, which can be regarded as the prediction of the to-be-coded frame. The synthesized frame is merged into reference picture list for motion estimation to further decrease the prediction residual. We integrate the proposed method into HM-16.20 under random access configuration. Experimental results show that the proposed method can significantly boost the coding performance, which provides 4.6% BD-rate reduction on average compared to HEVC baseline.