Harnessing consistency for improved test-time adaptation

Abstract

Test-time adaptation (TTA) is crucial for adjusting pre-trained models to new, unseen test data distributions without ground-truth labels, thereby addressing domain shifts commonly encountered in real-world scenarios. The most widely adopted self-training strategies in TTA include either pseudo-labeling or the minimization of prediction entropy. Different from these approaches, some research in natural language processing explored the use of consistency as a self-training objective. However, the performance improvements via consistency maximization have been limited. Based on this finding, we present a novel approach that employs consistency not as a primary self-training objective but as a metric for effective sample weighting and filtering. Our method, Consistency-TTA (CTTA), enhances performance and computational efficiency by implementing a sample weighting method that prioritizes samples demonstrating robustness to perturbations, and a sample filtering method that restricts backward pass to samples that are less prone to error accumulation. Our CTTA, which can be orthogonally combined with various state-of-the-art baselines, demonstrates performance improvements in extended adaptation tasks such as multi-modal TTA for 3D semantic segmentation and video domain adaptation. We evaluated CTTA on various corruption and natural domain shift datasets, consistently demonstrating meaningful performance improvements. Moreover, CTTA proved to be effective in both classification tasks and semantic segmentation benchmarks, such as CarlaTTA, highlighting its versatility across extended TTA applications.