Deep Learning in Video Stabilization Homography Estimation

Abstract

The main goal of digital video stabilization algorithms is to remove unwanted motion from a video sequence. The undesired motion is typically present in videos recorded by hand-held cameras, by cameras mounted on some moving platform (vehicle, boat, Unmanned Aerial Vehicle), or by stationary cameras under severe wind conditions. In this paper, the motion estimation step in video stabilization is performed in a novel way using deep learning homography matrix estimation. Convolutional Neural Network (CNN) takes two grayscale images as inputs, and produces a six degree of freedom affine transformation matrix that maps the pixels from the first image to the second one. After obtaining the homography transformation using a trained CNN, Kalman filter is used to separate the intentional from unintentional motion and calculate the final motion compensation transformation, stabilizing the video sequence.