Application Analysis of Hadoop in Big Data Processing

Abstract

How to use big data technology to effectively excavate and identify data information contained in user behaviors and further innovate services has become a development trend of the Internet big data. With the further increase of the amount of data, the configuration parameters involved further increase, and the optimization of configuration parameters has become the main bottleneck limiting the performance of MapReduce. Hadoop configuration involves many parameters, which have a great impact on the running jobs. These parameters just determine the overall performance of the cluster. This paper uses Hadoop technology to the Internet big data combining the optimization model. The construction process of Hadoop cluster environment is described in detail. Hadoop is applied to a file publishing system. For files of different orders of magnitude, the time-consuming operation of file upload is compared when the number of clusters is different. The experimental results show that the larger the amount of data and the number of cluster nodes, the stronger the ability of Hadoop cluster to process data. The results prove that this method effectively can solve the problems of the complex information of big data and improve the service efficiency for big data processing.