REP-Model: A deep learning framework for replacing ad billboards in soccer videos

Abstract

In this paper, we propose a novel framework for replacing advertisement contents in soccer videos with an automatic way by using deep learning strategies. We begin by applying UNET (an image segmentation convolutional neural network technique) for content segmentation and detection. Subsequently, after reconstructing the segmented content in the video frames (considering the apparent loss in detection), we will replace the unwanted content by new one using a homography mapping procedure. Furthermore, the replacement key points in each frame will be tracked into the next frames considering the camera zoom-in and zoom-out controlling. Since the movement of objects in video can disrupt the alignment between frames and correspondingly make the homography matrix calculation erroneous, we use Mask R-CNN to mask and remove the moving objects from the scene. Such framework is denominated as REP-Model which stands for a replacing model.