Hybrid data warehouse model for climate big data analysis

Abstract

The amount of data being collected and stored in the world is a highly unprecedented rate. The management and processing of huge data sets are time-consuming, costly, and hindrance to research. So, the process to store, manage, analyze and extract meaningful value from the vast volume of data is a big challenge to researchers. Data warehouse is a Decision Support System (DSS) technology that allows extracting, grouping and analyzing historical data from different sources in order to discover information relevant to decision making. Climate data is collected and stored in the national climatic data center (NCDC), the format of dataset support a rich set of meteorological elements. The data warehouse has the ability to manage data having a huge size in Terabytes range or higher, data is collected from different meteorological stations and stored in records to analyze it later in future. The process of big data analysis has become increasingly important for climate analysis field, which requires rapid and transparent data access. Recently, a new distributed computing paradigm, called MapReduce and it is implemented in an open source Hadoop, which has been widely adopted due to its impressive scalability and flexibility to handle structured, unstructured and semi-structured data. The purpose of this paper is to develop a conceptual data model and the implementation of hybrid data warehouse model to store NCDC's weather variables. The hybrid data warehouse model for climate big data enables the identification of weather patterns that would be useful for agriculture fields, climatic change studies and contingency plans over weather extreme conditions.