Innovation Method of Distributed Storage for Huge Data of Geological and Mineral Resources Based on Hadoop

Abstract

With the emergence of big data of TB and PB geological and mineral resources, the storage of large geological data has become a worldwide problem puzzling geologists. The traditional storage and service model of geological data is facing a great challenge. For example, when the scale of data increases dramatically, general relational database can not solve the problem of insufficient scalability, stability and efficiency of database system. In response to the above problems, this paper proposes a new method of geological and mineral data storage based on cloud computing environment combined with hadoop. Taking the mineral resources potential evaluation data of Chongqing as the research object, The proposed method in this paper is compared with the traditional Oracle database storage method in data storage experiments: (1) Small file optimization comparative experiment; (2) Hadoop and Oracle comparative experiment. The performance of writing operation, memory occupancy, data import and data export are tested in different way, and the comparison chart of performance is given. The experimental results show that the new storage method proposed in this paper is more efficient than the traditional method. At the same time, it effectively overcomes the problem of small file storage in Hadoop storage. The research results provide a new technical for the storage and management of geological and mineral data all over the country.