Column-Oriented Data Store for Astrophysical Data

Abstract

Astrophysical databases have used proprietary formats (especially the FITS format) to represent measured data and related metadata. The design of the FITS format was influenced by punch cards, thus it is extremely inappropriate for modern hardware. Even though this format is well established in the domain of astrophysics and will be definitely used as a common ground for data exchange, a new representation is required if the data are to be processed efficiently in a high performance manner. In this paper, we propose a specialized column-oriented format for the measured data, which allows much faster loading from the persistent data storage and direct application of the data in computational operations. Furthermore, we have tested various I/O methods implemented in modern operating systems to accommodate different access patterns that were observed in various use cases. We have created a prototype implementation of the proposed methods and experimentally evaluated their benefits.