Adaptive memory fusion for multi-frame optical flow estimation

Multi-frame optical flow models have demonstrated superior performance compared to two-frame models. Nevertheless, they suffer from several drawbacks, such as the absence of optical flow for the initial frames, the sharp increase in computational load as the number of input frames increases, and performance fluctuations when $\ge 3$ frames are incorporated. To address these issues, we propose AMFFlow, based on the Adaptive Memory Fusion (AMF) module, which effectively utilizes multi-frame information through memory fusion, thereby overcoming the aforementioned limitations. By employing a memory fusion mechanism, our method efficiently utilizes multi-frame information to estimate more accurate optical flow while maintaining a constant computational cost per frame. To our knowledge, AMFFlow is a pioneering model that can effectively exploit multi-frame information using $\ge 5$ frames for optical flow estimation. The AMF module is characterized by its low latency and plug-and-play capability. Additionally, we propose the Proportionally Debiased Endpoint Error (PDEPE), a novel metric designed to address evaluation errors caused by dataset bias in multi-frame optical flow models. Extensive experimental results demonstrate that AMFFlow achieves state-of-the-art (SOTA) generalization performance. Compared to the competing MemFlow model, the performance improvements on Sintel Training Clean and Sintel Training Final are approximately 5.9 % and 3.9 %, respectively. The source code and pre-trained models will be made publicly available at https://github.com/keacifer/AMFFlow.