PerfHD: Efficient ViT Architecture Performance Ranking using Hyperdimensional Computing

Abstract

Neural Architecture Search (NAS) aims at identifying the optimal network architecture for a specific need in an automated manner, which serves as an alternative to the manual process of model development, selection, evaluation and performance estimation. However, evaluating performance of candidate architectures in the search space during NAS, which often requires training and ranking a mass amount of architectures, is often prohibitively computation-demanding. To reduce this cost, recent works propose to estimate and rank the architecture performance with-out actual training or inference. In this paper, we present PerfHD, an efficient-while-accurate architecture performance ranking approach using hyperdimensional computing for the emerging vision transformer (ViT), which has demonstrated state-of-the-art (SOTA) performance in vision tasks. Given a set of ViT models, PerfHD can accurately and quickly rank their performance solely based on their hyper-parameters without training. We develop two encoding schemes for PerfHD, Gram-based and Record-based, to encode the features from candidate ViT architecture parameters. Using the VIMER-UFO benchmark dataset of eight tasks from a diverse range of domains, we compare PerfHD with four SOTA methods. Experimental results show that PerfHD can rank nearly 100K ViT models in about just 1 minute, which is up to 10X faster than SOTA methods, while achieving comparable or even superior ranking accuracy. We open-source PerfHD in PyTorch implementation at https://github.com/VU-DETAIL/PerfHD.