Deep-learning based web UI automatic programming

Abstract

The GUI building is an important part of web application development. Various studies such as WYSWYG web editor have been conducted to make this job convenient, where the job is composed of sketching of GUI and coding of HTML/CSS from the sketch. In this paper, we propose a novel way of web GUI building with computer vision and deep-learning. The proposed method requires only a hand-drawn sketch to build GUI. It recognizes web layout using computer vision algorithm, and web widgets using Faster R-CNN. With the recognized information, it makes HTML code automatically.