Generalized Spatio-Temporal Adaptive Normalization Framework

Abstract

In this paper, we propose Generalized Spatio-Temporal Adaptive Normalization (GSTAN) Framework for Generative Adversarial and Deep Learning Inference Architectures. By leveraging higher-order derivatives based temporal feature maps along with spatial feature map, our normalization approach leads to: (a) efficient generation of high-quality videos with better details and enhanced temporal coherence, and, (b) higher accuracy inference on multiple tasks. In order to evaluate model generalization, we performed experimental evaluation on multiple tasks including: video to video generation, video segmentation and activity recognition (classify the activity out of 101 activity classes, for a given input video). Detailed experimental analysis over a variety of datasets including CityScape, UCF101 and CK+ demonstrates superior performance of GSTAN and also provides the impact of its various configurations, including parallel GSTAN and sequential GSTAN.