MVFormer: Diversifying feature normalization and token mixing for efficient vision transformers

Abstract

Active research is currently underway to enhance the efficiency of vision transformers (ViTs). Most studies have focused solely on token mixers, overlooking the potential relationship with normalization. To boost diverse feature learning, we propose two components: multi-view normalization (MVN) and multi-view token mixer (MVTM). The MVN integrates three differently normalized features via batch, layer, and instance normalization using a learnable weighted sum, expected to offer diverse feature distribution to the token mixer, resulting in beneficial synergy. The MVTM is a convolution-based multiscale token mixer with local, intermediate, and global filters which incorporates stage specificity by configuring various receptive fields at each stage, efficiently capturing ranges of visual patterns. By adopting both in the MetaFormer block, we propose a novel ViT, multi-vision transformer (MVFormer). Our MVFormer outperforms state-of-the-art convolution-based ViTs on image classification with the same or lower parameters and MACs. Particularly, MVFormer variants, MVFormer-T, S, and B achieve 83.4 %, 84.3 %, and 84.6 % top-1 accuracy, respectively, on ImageNet-1 K benchmark.