CrossFlow: Learning cost volumes for optical flow by cross-matching local and non-local image features

Abstract

Optical flow is the pixel-level correspondence between two consecutive video frames. The cost volume plays an important role in deep learning-based optical flow methods. It measures the dissimilarity and the matching cost between two pixels in consecutive frames. Extensive optical flow methods have revolved around the cost volume. Most existing work constructs the cost volume by computing the dot product between the features of the target image and the source images, which is generally extracted by a shared convolutional neural network (CNN). However, these methods cannot adequately address long-standing challenges such as motion blur and large displacements. In this study, we propose the CrossFlow, computing the cost volume by cross-matching the local and non-local image features. The local and non-local features are extracted by the CNN and the transformer, respectively. Then, a total of four kinds of cost volumes are computed, and they are fused adaptively through a Softmax layer. As such, the final cost volume contains both the high- and low-frequency information. It facilitates the network in finding the correct correspondences from images with motion blur and large displacements. The experimental results demonstrate that our optical flow estimation method outperforms the baseline method (CRAFT) by 7% and 10% on the publicly available benchmarks Sintel and KITTI respectively, revealing the effectiveness of the proposed cost volume.