Role of Machine Learning in ETL Automation

Abstract

In the current business landscape, real-time analysis of enterprise data is very crucial for decision-makers of the organization to take strategic resolution and stay ahead of the competitors. Most of the time, it happens that data is outdated by the time it reaches to the user. The organization needs reliable, up to minute information to make better proactive business decisions, improve the process and organizational efficiency. Availability of information and business-critical report at real-time can be achieved through an automated ETL process. Typically, running a data warehouse in an enterprise requires coordination of many operations across many teams including applications and database teams. Also, it required a lot of manual intervention, which is error-prone. Executing all related steps in correct sequences under accurate conditions can be a challenge. Automated ETL process helps to address all these problems. Moreover, the preprocessing of data is a crucial step for making data ready to load in a data warehouse for analysis. Machine learning-based preprocessing can be used to ensure the quality of data. In this paper, we have addressed the issues faced in traditional data warehouse related to availability as well as the quality of data. We have explained how to automate the ETL process and how machine learning can be leveraged in the ETL process so that the quality and availability of data does not ever have been compromised and reached to the user on a near real-time basis.