A high performance multi-band data fusion approach for CSST based on column-oriented database

IF 1.9 4区物理与天体物理 Q2 ASTRONOMY & ASTROPHYSICS

Astronomy and Computing Pub Date : 2024-12-31 DOI:10.1016/j.ascom.2024.100922

Zhipeng Huang (黄智鹏) , Wei Du (杜薇) , Feng Wang (王锋) , Shoulin Wei (卫守林) , Hui Deng (邓辉) , Ying Mei (梅盈) , Tianmeng Zhang (张天萌)

{"title":"A high performance multi-band data fusion approach for CSST based on column-oriented database","authors":"Zhipeng Huang (黄智鹏) , Wei Du (杜薇) , Feng Wang (王锋) , Shoulin Wei (卫守林) , Hui Deng (邓辉) , Ying Mei (梅盈) , Tianmeng Zhang (张天萌)","doi":"10.1016/j.ascom.2024.100922","DOIUrl":null,"url":null,"abstract":"<div><div>High-performance multi-catalog fusion or cross-matching has always been an essential issue in astronomical data processing. In this study, we focus on the fusion of multi-band catalog data in a wide-area survey for the China Space Station Telescope (CSST). We implemented a simple and efficient data fusion method based on column-oriented database technology to produce a more consistent and accurate catalog, and this method can carry out the fusion of millions of source records in a few dozen seconds. We analyze and discuss several significant issues related to data fusion, such as the spatial partitioning and indexing of the target sky regions, the efficient implementation of fusion based on joining in the database, and the segmented processing method to address the issue of missing sources at different declinations. The performance profiling results show that by employing the MergeTree table engine within ClickHouse, establishing high-speed indexes based on the spatial partition index number, adopting an appropriate partitioning strategy, and maintaining orderly storage of records in the database according to the spatial partition index number, the efficient fusion of astronomical catalogs can be accomplished through SQL statements. Performance tests show that the proposed method can fulfill CSST data processing requirements, and it is also of reference value for future work related to massive astronomical data fusion. Compared with data fusion systems such as Large Survey DataBase (LSDB), our method can achieve similar performance results with consistent results.</div></div>","PeriodicalId":48757,"journal":{"name":"Astronomy and Computing","volume":"51 ","pages":"Article 100922"},"PeriodicalIF":1.9000,"publicationDate":"2024-12-31","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Astronomy and Computing","FirstCategoryId":"101","ListUrlMain":"https://www.sciencedirect.com/science/article/pii/S2213133724001379","RegionNum":4,"RegionCategory":"物理与天体物理","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"ASTRONOMY & ASTROPHYSICS","Score":null,"Total":0}

引用次数: 0

Abstract

High-performance multi-catalog fusion or cross-matching has always been an essential issue in astronomical data processing. In this study, we focus on the fusion of multi-band catalog data in a wide-area survey for the China Space Station Telescope (CSST). We implemented a simple and efficient data fusion method based on column-oriented database technology to produce a more consistent and accurate catalog, and this method can carry out the fusion of millions of source records in a few dozen seconds. We analyze and discuss several significant issues related to data fusion, such as the spatial partitioning and indexing of the target sky regions, the efficient implementation of fusion based on joining in the database, and the segmented processing method to address the issue of missing sources at different declinations. The performance profiling results show that by employing the MergeTree table engine within ClickHouse, establishing high-speed indexes based on the spatial partition index number, adopting an appropriate partitioning strategy, and maintaining orderly storage of records in the database according to the spatial partition index number, the efficient fusion of astronomical catalogs can be accomplished through SQL statements. Performance tests show that the proposed method can fulfill CSST data processing requirements, and it is also of reference value for future work related to massive astronomical data fusion. Compared with data fusion systems such as Large Survey DataBase (LSDB), our method can achieve similar performance results with consistent results.

查看原文本刊更多论文

基于列数据库的CSST高性能多波段数据融合方法

高性能的多星表融合或交叉匹配一直是天文数据处理中的关键问题。本文研究了中国空间站望远镜（CSST）广域巡天中多波段星表数据的融合。基于面向列的数据库技术，实现了一种简单高效的数据融合方法，生成更加一致和准确的目录，该方法可以在几十秒内完成数百万条源记录的融合。分析和讨论了数据融合的几个重要问题，如目标天空区域的空间划分和索引，基于数据库连接的融合的高效实现，以及解决不同赤纬缺失源问题的分割处理方法。性能分析结果表明，利用ClickHouse内部的MergeTree表引擎，根据空间分区索引号建立高速索引，采用合适的分区策略，并根据空间分区索引号维护数据库中记录的有序存储，可以通过SQL语句实现天文编目的高效融合。性能测试表明，该方法能够满足CSST数据处理要求，对未来海量天文数据融合相关工作具有参考价值。与大型调查数据库（Large Survey DataBase， LSDB）等数据融合系统相比，我们的方法可以获得相似的性能结果，且结果一致。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Astronomy and Computing ASTRONOMY & ASTROPHYSICSCOMPUTER SCIENCE,-COMPUTER SCIENCE, INTERDISCIPLINARY APPLICATIONS

CiteScore

4.10

自引率

8.00%

发文量

期刊介绍： Astronomy and Computing is a peer-reviewed journal that focuses on the broad area between astronomy, computer science and information technology. The journal aims to publish the work of scientists and (software) engineers in all aspects of astronomical computing, including the collection, analysis, reduction, visualisation, preservation and dissemination of data, and the development of astronomical software and simulations. The journal covers applications for academic computer science techniques to astronomy, as well as novel applications of information technologies within astronomy.