A Novel approach of GUI Mapping with image based widget detection and classification

Software testing is vital for the intellectual benefits of software reliability and quality. At present, Graphical user interfaces are the most common and widely used interfaces in the software industry. Furthermore, GUI Testing is an important approach to ensure the quality of software. Automated software testing is a GUI front end applications similar to APP, and WEB, etc and is a vastly time and resource-consuming task. Therefore, this will become even more complex in rapidly updated GUI applications such as Ex: Patches/Version updates of a mobile App, or the product and offer updates in marketing websites like Flipkart, Amazon, etc., in which the GUI components are continuously updated infinitely. Developing a test case scenario whenever a new GUI component is updated will affect the productivity of the application. In our experiment, we found a better way of improving GUI testing by consequently detecting and classifying GUI widgets using machine learning techniques. Additionally, we also found that detecting and classifying GUI objects in screenshots and reports with a position of the widgets (x, y coordinates) and type of the widgets, matches with trained samples, URL links, and screen links. Hence, we in this paper will analyze and devise an efficient automated testing strategy for Web Applications. This is a unique way of web Graphical user interface testing with a computer vision. This paper will also present the parameters used for object detection, classification, and evaluation with image processing using machine learning algorithms with better accuracy.