Scalable distributed storage for big scientific data

Abstract

In this paper we describe the distributed storage structure with indexing techniques which can be applied in scientific Big Data centers. Basic principal of this project is distributed (N, K)-block storage. We develop the descent of LH∗RS especially for multidimensional data arrays using multiscaled representation of these arrays and efficient pre-processing algorithms. Dataset is decomposed into data blocks of several levels using the Wavelet transform. The required dataset of any requested scale and resolution is reconstructed online from the corresponding set of downloaded blocks on client's side. In order to accelerate data queries processing we can additionally use a pre-computed statistic results blocks and their hierarchical representation. The main advantage of this approach is that we can use these results together with original data or even separately to serve different data queries with both value and dimension subsetting conditions. This approach can reduce the resource costs of corresponding scientific problems.