EarthDB: scalable analysis of MODIS data using SciDB

Earth scientists are increasingly experiencing difficulties with analyzing rapidly growing volumes of complex data. Those who must perform analysis directly on low-level National Aeronautics and Space Administration (NASA) Moderate Resolution Imaging Spectroradiometer (MODIS) Level 1B calibrated and geolocated data, for example, encounter an arcane, high-volume data set that is burdensome to make use of. Instead, Earth scientists typically opt to use higher-level "canned" products provided by NASA. However, when these higher-level products fail to meet the requirements of a particular project, a cruel dilemma arises: cope with data products that don't exactly meet the project's needs or spend an enormous amount of resources extracting what is needed from the unadulterated low-level data. In this paper, we present EarthDB, a system that eliminates this dilemma by offering the following contributions: 1. Enabling painless importing of MODIS Level 1B data into SciDB, a highly scalable science-oriented database platform that abstracts away the complexity of distributed storage and analysis of complex multi-dimensional data, 2. Defining a schema that unifies storage and representation of MODIS Level 1B data, regardless of its source file, 3. Supporting fast filtering and analysis of MODIS data through the use of an intuitive, high-level query language rather than complex procedural programming and, 4. Providing the ability to easily define and reconfigure entire analysis pipelines within the SciDB database, allowing for rapid ad-hoc analysis. To demonstrate this ability, we provide sample benchmarks for the construction of true-color (RGB) and Normalized Difference Vegetative Index (NDVI) images from raw MODIS Level 1B data using relatively simple queries with scalable performance.