ARSys: An Efficient and Cross-Platform Development, Deployment, and Runtime System for Mobile Augmented Reality

Abstract

Augmented reality (AR) offers users immersive experiences to interact with digital contents in their physical space. However, practical AR applications are challenged by the tight coupling of algorithm and engineering during the development and deployment phases as well as the execution requirements of hybrid AR subtasks on heterogeneous and resource-constraint mobile devices. In this work, we build an end-to-end, cross-platform, and efficient AR system, called ARSys. The infrastructure in ARSys adopts the new principle of integrated design, unifies and refines AR fundamental capabilities, supports streaming media processing, model inference, and real-time rendering by exposing high-performance tensor compute engine to top, and constructs a Python multi-instance virtual machine as the cross-platform AR task execution container. The runtime mechanism of ARSys schedules AR tasks in a pipeline parallelism way and allocates subtasks to hardware backends by optimizing the slowest node. The development workbench and the deployment platform in ARSys allow the decoupling of algorithms written in Python from engineering components in C/C++ and further support remote debugging and quick validation of AR algorithms. We extensively evaluate ARSys in practical AR applications across high-end, mid-end, and low-end Android and iOS devices, demonstrating higher development, deployment, and runtime efficiency than existing MediaPipe-oriented framework. ARSys has been integrated into Mobile Taobao for production use.