基于fastbit nosql数据库的NVST数据归档系统

IF 1.1 4区 物理与天体物理 Q3 ASTRONOMY & ASTROPHYSICS
Yingbo Liu, Feng Wang, Kaifan Ji, H. Deng, W. Dai, Bo Liang
{"title":"基于fastbit nosql数据库的NVST数据归档系统","authors":"Yingbo Liu, Feng Wang, Kaifan Ji, H. Deng, W. Dai, Bo Liang","doi":"10.5303/JKAS.2014.47.3.115","DOIUrl":null,"url":null,"abstract":"The New Vacuum Solar Telescope (NVST) is a 1-meter vacuum solar telescope that aims to observe the fine structures of active regions on the Sun. The main tasks of the NVST are high resolution imaging and spectral observations, including the measurements of the solar magnetic field. The NVST has been collecting more than 20 million FITS files since it began routine observations in 2012 and produces a maximum observational records of 120 thousand files in a day. Given the large amount of files, the effective archiving and retrieval of files becomes a critical and urgent problem. In this study, we implement a new data archiving system for the NVST based on the Fastbit Not Only Structured Query Language (NoSQL) database. Comparing to the relational database (i.e., MySQL; My Structured Query Language), the Fastbit database manifests distinctive advantages on indexing and querying performance. In a large scale database of 40 million records, the multi-field combined query response time of Fastbit database is about 15 times faster and fully meets the requirements of the NVST. Our study brings a new idea for massive astronomical data archiving and would contribute to the design of data management systems for other astronomical telescopes.","PeriodicalId":49994,"journal":{"name":"Journal of the Korean Astronomical Society","volume":"47 1","pages":"115-122"},"PeriodicalIF":1.1000,"publicationDate":"2014-06-30","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"7","resultStr":"{\"title\":\"NVST DATA ARCHIVING SYSTEM BASED ON FASTBIT NOSQL DATABASE\",\"authors\":\"Yingbo Liu, Feng Wang, Kaifan Ji, H. Deng, W. Dai, Bo Liang\",\"doi\":\"10.5303/JKAS.2014.47.3.115\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"The New Vacuum Solar Telescope (NVST) is a 1-meter vacuum solar telescope that aims to observe the fine structures of active regions on the Sun. The main tasks of the NVST are high resolution imaging and spectral observations, including the measurements of the solar magnetic field. The NVST has been collecting more than 20 million FITS files since it began routine observations in 2012 and produces a maximum observational records of 120 thousand files in a day. Given the large amount of files, the effective archiving and retrieval of files becomes a critical and urgent problem. In this study, we implement a new data archiving system for the NVST based on the Fastbit Not Only Structured Query Language (NoSQL) database. Comparing to the relational database (i.e., MySQL; My Structured Query Language), the Fastbit database manifests distinctive advantages on indexing and querying performance. In a large scale database of 40 million records, the multi-field combined query response time of Fastbit database is about 15 times faster and fully meets the requirements of the NVST. Our study brings a new idea for massive astronomical data archiving and would contribute to the design of data management systems for other astronomical telescopes.\",\"PeriodicalId\":49994,\"journal\":{\"name\":\"Journal of the Korean Astronomical Society\",\"volume\":\"47 1\",\"pages\":\"115-122\"},\"PeriodicalIF\":1.1000,\"publicationDate\":\"2014-06-30\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"7\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of the Korean Astronomical Society\",\"FirstCategoryId\":\"101\",\"ListUrlMain\":\"https://doi.org/10.5303/JKAS.2014.47.3.115\",\"RegionNum\":4,\"RegionCategory\":\"物理与天体物理\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q3\",\"JCRName\":\"ASTRONOMY & ASTROPHYSICS\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of the Korean Astronomical Society","FirstCategoryId":"101","ListUrlMain":"https://doi.org/10.5303/JKAS.2014.47.3.115","RegionNum":4,"RegionCategory":"物理与天体物理","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q3","JCRName":"ASTRONOMY & ASTROPHYSICS","Score":null,"Total":0}
引用次数: 7

摘要

新型真空太阳望远镜(NVST)是一个1米长的真空太阳望远镜,旨在观察太阳活跃区域的精细结构。NVST的主要任务是高分辨率成像和光谱观测,包括太阳磁场的测量。自2012年开始常规观测以来,NVST已经收集了超过2000万个FITS文件,每天产生的最大观测记录为12万个文件。面对海量的文件,如何对文件进行有效的归档和检索就成为一个关键而迫切的问题。在本研究中,我们基于Fastbit Not Only Structured Query Language (NoSQL)数据库实现了一个新的NVST数据归档系统。与关系型数据库(即MySQL;我的结构化查询语言),Fastbit数据库在索引和查询性能上表现出独特的优势。在4000万条记录的大型数据库中,Fastbit数据库的多字段组合查询响应时间要快15倍左右,完全满足NVST的要求。本研究为海量天文数据的归档提供了新的思路,对其他天文望远镜数据管理系统的设计也有一定的借鉴意义。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
NVST DATA ARCHIVING SYSTEM BASED ON FASTBIT NOSQL DATABASE
The New Vacuum Solar Telescope (NVST) is a 1-meter vacuum solar telescope that aims to observe the fine structures of active regions on the Sun. The main tasks of the NVST are high resolution imaging and spectral observations, including the measurements of the solar magnetic field. The NVST has been collecting more than 20 million FITS files since it began routine observations in 2012 and produces a maximum observational records of 120 thousand files in a day. Given the large amount of files, the effective archiving and retrieval of files becomes a critical and urgent problem. In this study, we implement a new data archiving system for the NVST based on the Fastbit Not Only Structured Query Language (NoSQL) database. Comparing to the relational database (i.e., MySQL; My Structured Query Language), the Fastbit database manifests distinctive advantages on indexing and querying performance. In a large scale database of 40 million records, the multi-field combined query response time of Fastbit database is about 15 times faster and fully meets the requirements of the NVST. Our study brings a new idea for massive astronomical data archiving and would contribute to the design of data management systems for other astronomical telescopes.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
Journal of the Korean Astronomical Society
Journal of the Korean Astronomical Society 地学天文-天文与天体物理
CiteScore
1.30
自引率
10.00%
发文量
0
审稿时长
>12 weeks
期刊介绍: JKAS is an international scientific journal publishing papers in all fields of astronomy and astrophysics. All manuscripts are subject to the scrutiny of referees. Manuscripts submitted to JKAS must comply with the ethics policy of JKAS. Six regular issues are published each year on February 28, April 30, June 30, August 31, October 31, and December 31. One year''s issues compose one volume.
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信