Image Segmentation based Background Removal and Replacement

Abstract

Privacy is a great cause of concern. As many of us are concerned and self-conscious about what is visible in our background while having an online meeting. To provide some privacy during an online meeting our main objective is to blur/remove/replace the background of a live webcam video stream using a machine learning model trained to segment out persons in an image, whose results are then refined using image processing techniques. Currently, it is available in online meeting apps like Zoom, Google Meet, Microsoft Teams, etc., but there is no open-source code available for it so we also want to make an online repository, from where anyone can take our code, and implement it in their project. Our program uses a webcam on a computer and proceeds in the following way. First, we capture a live frame from the webcam, and then pre-process it, so that it is suitable for supplying as input to the machine-learning model. This provides us with a rough predicted foreground mask. Then, to refine it, we perform morphological closing, which is used to remove small imperfections in the mask. We then obtain the outline of the detected person using contour finding and fill its interior region with white by using a flood-fill operation. After all, of this, we have generated a foreground mask layer in an alpha channel whose white area shows the foreground, which we have to leave as it, is and the black area shows background, which we have to change. Therefore, we combine this generated mask along with its corresponding frame and, optionally, a background image, to generate its corresponding frame. Repeatedly doing this for every frame we receive from the webcam continuously, we have a live video with edited background with a decent frame rate of 26-28 FPS.