Storage optimisation and distributed architecture for time series reconstruction of massive astronomical catalogues

Abstract

Time series reconstruction of astronomical catalogues is an important part of data archiving and a basis for time-domain astronomical analysis in the era of time-domain astronomy. As the field of view and sampling frequency of various time-domain telescopes increase, the amount of data to be processed becomes larger and larger. How to optimize the spatial and temporal efficiency of this process with the aid of computer technology becomes a hot issue. To address the problem of spatial efficiency, in this paper, we propose a time series data compression algorithm based on the negative database and dynamic programming, and on this basis, we design a multi-level storage and access query architecture for hot data and non-hot data, which greatly compresses the storage space of data while ensuring the query efficiency. To address the issue of time efficiency, this paper proposes a spatio-temporal data partitioning and layout algorithm suitable for distributed architecture, whose nested round-robin strategy has a wide range of load balancing effects on different spatial locations, temporal locations, and different ranges of temporal data queries, which can effectively ensure the execution efficiency of the distributed system. Experimental results show that the proposed optimization algorithm can keep the system at a low load skewness level of about 4% and save about 83% of storage space.