Real-time Inferencing and Training of Artificial Neural Network for Adaptive Latency Negation in Distributed Virtual Environments

Abstract

With recent trends to move more computation and applications to the cloud, latency is one of the major factors that can degrade the experience. This is even more true when it comes to distributed virtual environments (DVE) where users are interacting in real-time with a server-side environment. Past works have attempted to use user motion prediction to negate the effects of latency in VR systems but often struggle with long prediction lengths or fast user motion speed. In our previous work, we found that using an artificial neural network (ANN) for user motion prediction was feasible; however, our test showed relatively simple ANNs struggled when the latency varied, or user behavior changed drastically. To combat these issues, we are proposing the use of real-time inferencing and training (RTIT) to give our stacked LSTM the capability to adapt. By utilizing RTIT, our model is able to maintain a low prediction error when the system experiences both various amounts of latency and different interaction patterns. In addition, as the latency and user motion speed rise, our method remains robust longer than polynomial regression-based predictors.