Data abstraction through density estimation by storage management

One way to cope with the constantly growing amount of scientific data to be analyzed is to derive data abstractions from the original data. Data abstractions can provide a representation of the data in compressed form where the data's semantic structure is maintained. The author has explored data abstractions based on density estimation. The method to estimate the density of scientific data sets is based on the directory of a multidimensional data access structure. This data density estimator is called directory estimator. It is based on multidimensional adaptive histograms and is therefore computationally efficient, even for large data sets and many dimensions. The paper describes the methodology in general and focuses on the estimator's accuracy in particular. The accuracy of the directory estimator depends on the parameters of the access structures used, such as the bucket capacity. She evaluates the choice of bucket capacity theoretically as well as empirically with the ISE (integrated squared error) being the measure of error and using a grid file as the data access structure. A useful application of the directory estimator in the field of scientific data is presented with a practical example from astronomy.