UI Components Recognition System Based On Image Understanding

Abstract

Before the release of mobile application products, a lot of repeated testing is often required. In the process of mobile application testing, the core problem is to locate the UI components on the mobile application screenshots. There are many methods to automatically identify UI components, but in some cases, such as crowdsourcing testing, it is difficult to use automatic methods to identify UI components. In view of this, the APP UI components recognition system based on image understanding provides new solutions and methods for application scenarios that are difficult to automatically locate components. We investigate Android UI component information, use image understanding analysis to extract component images on screenshot, design and implement a convolutional neural networks, and then use trained CNN to classify these images. The classification accuracy is up to 96.97%. In the end, we get the component information contained in screenshot.