Yingbo Liu, Feng Wang, Kaifan Ji, H. Deng, W. Dai, Bo Liang
{"title":"基于fastbit nosql数据库的NVST数据归档系统","authors":"Yingbo Liu, Feng Wang, Kaifan Ji, H. Deng, W. Dai, Bo Liang","doi":"10.5303/JKAS.2014.47.3.115","DOIUrl":null,"url":null,"abstract":"The New Vacuum Solar Telescope (NVST) is a 1-meter vacuum solar telescope that aims to observe the fine structures of active regions on the Sun. The main tasks of the NVST are high resolution imaging and spectral observations, including the measurements of the solar magnetic field. The NVST has been collecting more than 20 million FITS files since it began routine observations in 2012 and produces a maximum observational records of 120 thousand files in a day. Given the large amount of files, the effective archiving and retrieval of files becomes a critical and urgent problem. In this study, we implement a new data archiving system for the NVST based on the Fastbit Not Only Structured Query Language (NoSQL) database. Comparing to the relational database (i.e., MySQL; My Structured Query Language), the Fastbit database manifests distinctive advantages on indexing and querying performance. In a large scale database of 40 million records, the multi-field combined query response time of Fastbit database is about 15 times faster and fully meets the requirements of the NVST. Our study brings a new idea for massive astronomical data archiving and would contribute to the design of data management systems for other astronomical telescopes.","PeriodicalId":49994,"journal":{"name":"Journal of the Korean Astronomical Society","volume":null,"pages":null},"PeriodicalIF":1.1000,"publicationDate":"2014-06-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"7","resultStr":"{\"title\":\"NVST DATA ARCHIVING SYSTEM BASED ON FASTBIT NOSQL DATABASE\",\"authors\":\"Yingbo Liu, Feng Wang, Kaifan Ji, H. Deng, W. Dai, Bo Liang\",\"doi\":\"10.5303/JKAS.2014.47.3.115\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"The New Vacuum Solar Telescope (NVST) is a 1-meter vacuum solar telescope that aims to observe the fine structures of active regions on the Sun. The main tasks of the NVST are high resolution imaging and spectral observations, including the measurements of the solar magnetic field. The NVST has been collecting more than 20 million FITS files since it began routine observations in 2012 and produces a maximum observational records of 120 thousand files in a day. Given the large amount of files, the effective archiving and retrieval of files becomes a critical and urgent problem. In this study, we implement a new data archiving system for the NVST based on the Fastbit Not Only Structured Query Language (NoSQL) database. Comparing to the relational database (i.e., MySQL; My Structured Query Language), the Fastbit database manifests distinctive advantages on indexing and querying performance. In a large scale database of 40 million records, the multi-field combined query response time of Fastbit database is about 15 times faster and fully meets the requirements of the NVST. Our study brings a new idea for massive astronomical data archiving and would contribute to the design of data management systems for other astronomical telescopes.\",\"PeriodicalId\":49994,\"journal\":{\"name\":\"Journal of the Korean Astronomical Society\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":1.1000,\"publicationDate\":\"2014-06-30\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"7\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of the Korean Astronomical Society\",\"FirstCategoryId\":\"101\",\"ListUrlMain\":\"https://doi.org/10.5303/JKAS.2014.47.3.115\",\"RegionNum\":4,\"RegionCategory\":\"物理与天体物理\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q3\",\"JCRName\":\"ASTRONOMY & ASTROPHYSICS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of the Korean Astronomical Society","FirstCategoryId":"101","ListUrlMain":"https://doi.org/10.5303/JKAS.2014.47.3.115","RegionNum":4,"RegionCategory":"物理与天体物理","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"ASTRONOMY & ASTROPHYSICS","Score":null,"Total":0}
引用次数: 7
摘要
新型真空太阳望远镜(NVST)是一个1米长的真空太阳望远镜,旨在观察太阳活跃区域的精细结构。NVST的主要任务是高分辨率成像和光谱观测,包括太阳磁场的测量。自2012年开始常规观测以来,NVST已经收集了超过2000万个FITS文件,每天产生的最大观测记录为12万个文件。面对海量的文件,如何对文件进行有效的归档和检索就成为一个关键而迫切的问题。在本研究中,我们基于Fastbit Not Only Structured Query Language (NoSQL)数据库实现了一个新的NVST数据归档系统。与关系型数据库(即MySQL;我的结构化查询语言),Fastbit数据库在索引和查询性能上表现出独特的优势。在4000万条记录的大型数据库中,Fastbit数据库的多字段组合查询响应时间要快15倍左右,完全满足NVST的要求。本研究为海量天文数据的归档提供了新的思路,对其他天文望远镜数据管理系统的设计也有一定的借鉴意义。
NVST DATA ARCHIVING SYSTEM BASED ON FASTBIT NOSQL DATABASE
The New Vacuum Solar Telescope (NVST) is a 1-meter vacuum solar telescope that aims to observe the fine structures of active regions on the Sun. The main tasks of the NVST are high resolution imaging and spectral observations, including the measurements of the solar magnetic field. The NVST has been collecting more than 20 million FITS files since it began routine observations in 2012 and produces a maximum observational records of 120 thousand files in a day. Given the large amount of files, the effective archiving and retrieval of files becomes a critical and urgent problem. In this study, we implement a new data archiving system for the NVST based on the Fastbit Not Only Structured Query Language (NoSQL) database. Comparing to the relational database (i.e., MySQL; My Structured Query Language), the Fastbit database manifests distinctive advantages on indexing and querying performance. In a large scale database of 40 million records, the multi-field combined query response time of Fastbit database is about 15 times faster and fully meets the requirements of the NVST. Our study brings a new idea for massive astronomical data archiving and would contribute to the design of data management systems for other astronomical telescopes.
期刊介绍:
JKAS is an international scientific journal publishing papers in all fields of astronomy and astrophysics. All manuscripts are subject to the scrutiny of referees. Manuscripts submitted to JKAS must comply with the ethics policy of JKAS. Six regular issues are published each year on February 28, April 30, June 30, August 31, October 31, and December 31. One year''s issues compose one volume.