Improving Aerospace Big Data Infrastructure and Applications with Distributed File System and Massive Parallel Calculation

Abstract

As the aerospace business growing rapidly, data flow and volume has exploded in recent years, bringing chances and challenges to big data infrastructures and applications in this field. In traditional aerospace data and application centers, data is stored in network attached storages(NAS) and processed by sequential or low level parallel programs, which can hardly meet the demand of performance, availability and scalability. In this paper, we provided a big data infrastructure based on HDFS for big data centers, which can improve the availability and scalability remarkably. Besides, we gather a series of typical big data applications in aerospace filed as benchmarks, analyzes their characteristics and accelerates them in MapReduce framework. The experiment result shows that among all the benchmarks, the speedup is 4.98 to the peak and 3.87 on the average.