Boosting representation diversity in video transformers via segmented contrastive masked autoencoders

Abstract

In self-supervised representation learning, contrastive learning leverages data invariance to identify global patterns; however, its lack of local constraints often results in video representations that are overly focused on global content, thereby limiting feature diversity. To address this, we propose Segmented Contrastive Masked Autoencoders (SCMA). By extending the masked distillation paradigm to the spatiotemporal domain, our framework integrates contrastive learning with masked autoencoders via teacher-student self-distillation. SCMA enhances the capability of contrastive learning to capture local details in videos, effectively improving the diversity of video representations learned by ViT models. Furthermore, we introduce a Segmented Mask Sampling strategy to partition videos uniformly and apply adaptive masking, enhancing both training efficiency and robustness to input variations. Experimental results demonstrate that SCMA achieves superior top-1 accuracy on a range of video action recognition benchmarks under both fine-tuning and linear probing settings, while also demonstrating strong performance in video object segmentation.