Research on Parallel LSTM Algorithm Based on Spark

Abstract

Aiming at the problems of large amount of data collected by airborne sensors, lack of data association, and low processing efficiency, this paper proposes a parallel LSTM algorithm model suitable for Spark platform. First, use the Spark platform to complete the traversal scan operation in the memory RDD of all nodes in the distributed cluster, and combine the directed acyclic graph to create a Pipeline pipeline to implement a parallel computing framework. An algorithm model to optimize the parameters of LSTM neural network is proposed, and load balancing processing method is introduced to realize that all nodes of the distributed system can share the computing tasks in a balanced manner. The experimental results show that compared to the stand-alone case, the parallelized LSTM algorithm improves the efficiency. The prediction efficiency of the LSTM algorithm model after load balancing processing is higher, which shows that the distribution of traversal tasks of each node is more balanced and the degree of parallelization is higher.