A Novel Big Data Processing Approach to Feature Extraction for Electrical Discharge Machining based on Container Technology

EDM (Electrical Discharge Machining) is a process to remove metal from conductive materials using electrical sparks. To monitor the EDM process using virtual metrology (VM), we need to obtain the electrode’s voltage and current signals of a machine tool. Due to the nature of EDM, the sensors installed on the machine tool acquire the signals at a high sampling rate and generate a vast amount of data in a short time, thereby raising the big-data processing issue. Our previous work proposed an efficient approach called BEDPS to process the EDM big data in a Hadoop distributed cluster. This paper presents a novel big data processing approach to feature extraction for EDM by using container technology (i.e., Docker and Kubernetes). We re-implement some Spark algorithms of BEDPS in Python (originally in Scala) and then run the refined BEDPS in containers in a Kubernetes cluster. Testing results show that the refined BEDPS developed in this study can reduce the execution time by almost half, compared to the original Scala version (9.6577 minutes vs. 19.2735 minutes). The adoption of Python in Spark is also shown to have similar performance with Scala, although there are some cases where Python performance falls short, for example, parallel processing using Python parallel processing library. The results also show that the Kubernetes cluster is promising to be an alternative way, other than the Hadoop, for processing big data. At the same time, it can bring some advantages to the big data processing applications, such as easy deployment, robustly running, load balance, self-healing, failover, and horizontal auto-scaling for containerized applications.