A ‘deep’ review of video super-resolution

Abstract

Video super-resolution (VSR) is an ill-posed inverse problem where the goal is to obtain high-resolution video content from a low-resolution counterpart. In this survey, we trace the history of video super-resolution techniques beginning with traditional methods, showing the evolution towards techniques that use shallow networks and finally, the recent trends where deep learning algorithms result in state-of-the-art performance. Specifically, we consider 60 neural network-based VSR techniques in addition to 8 traditional VSR techniques. We extensively cover the deep learning-based techniques including the latest models and introduce a novel taxonomy depending on their architecture. We discuss the pros and cons of each category of techniques. We consider the various components of the problem including the choice of loss functions, evaluation criteria and the benchmark datasets used for evaluation. We present a comparison of the existing techniques using common datasets, providing insights into the relative rankings of these methods. We compare the network architectures based on their computation speed and the network complexity. We also discuss the limitations of existing loss functions and the evaluation criteria that are currently used and propose alternate suggestions. Finally, we identify some of the current challenges and provide future research directions towards video super-resolution, thus providing a comprehensive understanding of the problem.