Multi-scale spatio-temporal memory network for semi-supervised video object segmentation

Abstract

Recently, video object segmentation has received great attention in the computer vision community. Most of the existing methods exhibit poor segmentation performance when dealing with complex scenes with local information confusion such as foreground-foreground similarity and foreground-background similarity. To tackle this problem, this study proposes a semi-supervised video target segmentation (semi-VOS) framework multi-scale memory (MSM) based on spatio-temporal memory, aiming to solve the problem of insufficient local attention by constructing the correlation of space and time in a given video. The spatio-temporal memory network is used as the basic framework to display and store the target appearance feature calculated from the historical frames in the external memory, resulting in employing the historical frame feature information over a long period of time. Considering the difficulty of computing local correlations, a filtering mechanism is designed to remove the global noise in the memory reading stage. An atrous spatial pyramid pooling module is added before decoding to prevent the local information loss induced by the downsampling operation. Extensive experiments are conducted on video object segmentation benchmarks including DAVIS-16 validation, DAVIS-17 validation and test, and YouTube-2018 validation datasets. The experimental results demonstrate the feasibility and effectiveness of the proposed framework on various complex scenarios compared with previous methods.