DVC2: Deep video cascade clustering from video structures

Video clustering is a critical unsupervised learning task, where category labels are unavailable, unlike in supervised video classification. The primary challenge is learning meaningful video representations without annotations to effectively group similar videos. Most existing methods extract frame-level features and apply standard clustering algorithms such as K-means, but they often fail to capture temporal relationships inherent in video data. In this paper, we introduce Deep Video Cascade Clustering (${\text{DVC}}^2$), a novel unsupervised video learning paradigm. Unlike image-based clustering methods, ${\text{DVC}}^2$ first learns an initial video representation through frame clustering, which serves as guidance, and then aligns video clustering results with both long-term and short-term structures as well as nearest neighbors. We evaluate ${\text{DVC}}^2$ on benchmark datasets, including UCF101 and Kinetics-400, achieving state-of-the-art results. Notably, even in annotation-free scenarios where self-supervised learning with K-means already yields reasonable clustering, ${\text{DVC}}^2$ demonstrates significantly superior performance.