A Smart Automatic Agent Operating System Using Screen Analysis And Remote Control

Abstract

In this century, more automated production has been driven by high demand. As a result, more automated production machines are required to meet market demands. Unfortunately, most machine screen activities still need humans to control manually, and most of the operations are simple and repeated. The human eye must constantly assess the production situation and control the keyboard and mouse to operate machinery to do excessively repetitive tasks. We proposed a smart automated agent operating system called “iScript” to upgrade the old machine to a more automated machine. “iScript” is a software solution that uses image recognition techniques to automate agent operation by automatically controlling the keyboard and mouse to perform agent operations using preset scripts. The agent operation system is also integrated with a modular software design. The essential modules include the Steps Control Module (SCM), Function Plug-in Module (FPM), Variable Module (VM), Condition Control Module (CCM), and Operator Control Module (OCM). The modular design makes the script setting highly expandable. It can set customized scripts for varied purposes and applications dynamically, making production process management more accessible and efficient and accomplishing many tiresome tasks necessary for industrial automation upgrades. The proposed smart automatic agent operating system has been applied to several industrial applications, and the cases of application are shared in this research.