Persival: Simulating Complex 3D Meshes on Resource-Constrained Mobile AR Devices Using Interpolation

Abstract

Simulations are an important part of analyzing and understanding systems, including not only technical but also bio-mechanical subjects such as the musculoskeletal apparatus of the human body. Detailed, biophysical simulations are complex and require a substantial amount of computational resources. With the advent of mobile AR devices such as the Microsoft HoloLens, new challenges arise to run or represent the results of such complex simulations on resource-constrained devices. In this paper we propose a deep-learning-based mobile simulation approach for the contraction of a human muscle model on an AR device (MS HoloLens 2). To elaborate, we present a two-step workflow consisting of simulating the deformation of the 3D geometry of the biceps, of which a subset of points can be interpolated back to full resolution. This allows to either offload the full simulation, just communicating the subset of nodal points, or to use a lower-quality local simulation restricted to the subset. Interpolation is done locally in both cases. The interpolation model consists of a dense, single hidden layer neural network. A mesh simplification method is combined with a genetic algorithm to determine the optimal subset of mesh nodes to interpolate from. In purely local execution, our simulation and interpolation model is able to accurately predict the position of 2809 nodal points based on as few as 30, while using 97.78 % less energy and evaluating up to 1.23 times faster compared to the local reference model. In an ideal distributed scenario energy consumption decreases by 99 % and evaluation time is up to 32.42 times faster. For the latter, it also reduces communication-data to 1.2 % of the full resolution mesh.