Load Balancing Optimization for Transformer in Distributed Environment

In recent years, the demand for artificial intelligence applications has increased dramatically. Complex models can promote machine learning to achieve excellent results, but computing efficiency has gradually reached a bottleneck. Therefore, more researchers are exploring the improvement of the efficiency of intelligent computing systems. Distributed machine learning can improve the efficiency of model training and inference, but problems such as communication delay and load imbalance between computing nodes still exist. In the multi-GPU distributed computing environment, this paper takes the vision field algorithm VIT (vision transformer) as the optimization object, which has the advantage of convenient parallel training, and proposes several related solutions. Firstly, the parameter server is used as the system logic architecture and in order to reduce the idleness of the computing devices during the training process, the device working status query mechanism is designed to realize load balancing. Secondly, combined with the pre-trained small VIT algorithm model, semi-asynchronous communication method is proposed to reduce the communication overhead of computing devices and accelerate global convergence. The results of this experiment carried out in the existing distributed environment has demonstrated that compared with the existing synchronization method, the computational efficiency has been improved well under the premise of slightly reducing the accuracy.