SMG-MATSM: Scene Memory Generation Based on Motion-Aware Temporal Style Modulation

Abstract

Scene memory generation (SMG) refers to training AI agents to recall scene memories similarly to the human brain. This is the key work to realize the artificial memory system. The challenge is to generate scenes rich in motion and keep it realistic while ensuring temporal consistency. Inspired by the principles of memory function in brain neuroscience, this paper proposes a motion-aware scene generation model named SMG based on motion-aware temporal style modulation (SMG-MATSM), which ensures temporal consistency by redesigning the temporal latent representation and constructing a motion matrix to guide the motion of intermediate latent variables. The motion matrix preserves motion consistency in the scene memory through both the cosine similarity and the Mahalanobis distance of intermediate latent variables of adjacent frames. Additionally, SMG-MATSM uses a style-based approach and enhances conditional features through the motion matrix during the scene memory synthesis process. Experimental results show that SMG-MATSM has better effect of action-enriched scene memory generation, and has varying degrees of efficiency improvement on different datasets with Frechet video distance and Frechet inception distance evaluation metrics.