J. Bakken, E. Berman, Chih-Hao Huang, A. Moibenko, D. Petravick, M. Zalokar
{"title":"The Fermilab data storage infrastructure","authors":"J. Bakken, E. Berman, Chih-Hao Huang, A. Moibenko, D. Petravick, M. Zalokar","doi":"10.1109/MASS.2003.1194845","DOIUrl":"https://doi.org/10.1109/MASS.2003.1194845","url":null,"abstract":"Fermilab, in collaboration with the DESY laboratory in Hamburg, Germany, has created a petabyte scale data storage infrastructure to meet the requirements of experiments to store and access large data sets. The Fermilab data storage infrastructure consists of the following major storage and data transfer components: the Enstore mass storage system, DCache distributed data cache, FTP and grid FTP for primarily external data transfers. This infrastructure provides a data throughput sufficient for transferring data from experimental data acquisition systems. It also allows access to data in the grid framework.","PeriodicalId":285161,"journal":{"name":"20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, 2003. (MSST 2003). Proceedings.","volume":"5 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2003-04-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134374070","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
H. Thompson, C. Tilmes, R. Cavey, Bill Fink, P. Lang, B. Kobler
{"title":"Considerations and performance evaluations of shared storage area networks at NASA Goddard Space Flight Center","authors":"H. Thompson, C. Tilmes, R. Cavey, Bill Fink, P. Lang, B. Kobler","doi":"10.1109/MASS.2003.1194850","DOIUrl":"https://doi.org/10.1109/MASS.2003.1194850","url":null,"abstract":"The NASA Goddard Space Flight Center (GSFC) in Greenbelt Maryland is exploring advanced storage architectures for retaining and distributing its large data holdings. As a research vehicle, a multi-building Storage Area Network (SAN) was deployed at GSFC in early 2002. The initial objective was to demonstrate the feasibility and advantages of fibre channel-connected, centralized storage as it applies to a campus installation. The secondary objective was to illustrate the advantages of a heterogeneous SAN shared file system that would allow a single instance of data to be globally shared amongst multiple SAN-connected clients on different platforms. The GSFC SAN has since been extended to include off-campus connections for evaluating the Internet Protocol (IP) as an option for connecting a broader, more geographically dispersed user base. The focus of this paper is the series of tests conducted to characterize distance data sharing using both native FC and IP based technologies. These experiments include both standard I/O benchmarks as well as representative GSFC applications.","PeriodicalId":285161,"journal":{"name":"20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, 2003. (MSST 2003). Proceedings.","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2003-04-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129838086","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Efficient metadata management in large distributed storage systems","authors":"S. Brandt, E. L. Miller, D. Long, Lan Xue","doi":"10.1109/MASS.2003.1194865","DOIUrl":"https://doi.org/10.1109/MASS.2003.1194865","url":null,"abstract":"Efficient metadata management is a critical aspect of overall system performance in large distributed storage systems. Directory subtree partitioning and pure hashing are two common techniques used for managing metadata in such systems, but both suffer from bottlenecks at very high concurrent access rates. We present a new approach called lazy hybrid (LH) metadata management that combines the best aspects of these two approaches while avoiding their shortcomings.","PeriodicalId":285161,"journal":{"name":"20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, 2003. (MSST 2003). Proceedings.","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2003-04-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131135883","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
A. Rajasekar, M. Wan, Reagan Moore, G. Kremenek, Tom Guptil
{"title":"Data grids, collections, and grid bricks","authors":"A. Rajasekar, M. Wan, Reagan Moore, G. Kremenek, Tom Guptil","doi":"10.1109/MASS.2003.1194830","DOIUrl":"https://doi.org/10.1109/MASS.2003.1194830","url":null,"abstract":"Data grids federate storage resources. They provide a logical name space that can be used to register digital entities, a storage repository abstraction for manipulating data, and a high level abstraction for supporting user-selected interfaces. Data grids can be used to build persistent collections. Data can be stored across multiple types of storage systems with persistent copies kept in archives. Persistent identifiers can be kept in the logical name space. By integrating grid bricks (commodity based disk caches) with archival storage systems, one can assemble a data management environment that supports both interactive access (data picking) and long-term persistent storage. Examples of the creation of interactive data picking environments will be given that integrate grid brick technology with large-scale archives.","PeriodicalId":285161,"journal":{"name":"20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, 2003. (MSST 2003). Proceedings.","volume":"62 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2003-04-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114703602","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Implementing and evaluating jukebox schedulers using JukeTools","authors":"M. Lijding, S. Mullender, P. Jansen","doi":"10.1109/MASS.2003.1194843","DOIUrl":"https://doi.org/10.1109/MASS.2003.1194843","url":null,"abstract":"Scheduling jukebox resources is important to build efficient and flexible hierarchical storage systems. JukeTools is a toolbox that helps in the complex tasks of implementing and evaluating jukebox schedulers. It allows the fast development of jukebox schedulers. The schedulers can be tested in numerous environments, both real and simulated types. JukeTools helps the developer to easily detect errors in the schedules. Analyzer tools create detailed reports on the behavior and performance of any of the scheduler, and provide comparisons between different schedulers. This paper describes the functionality offered by JukeTools, with special emphasis on how the toolbox can be used to develop jukebox schedulers.","PeriodicalId":285161,"journal":{"name":"20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, 2003. (MSST 2003). Proceedings.","volume":"11 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2003-04-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114460173","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"NCDC the \"one stop shop\" for all WSR-88D level II data services","authors":"S.A. Del Greco","doi":"10.1109/MASS.2003.1194847","DOIUrl":"https://doi.org/10.1109/MASS.2003.1194847","url":null,"abstract":"The National Climatic Data Center (NCDC) currently receives Weather Surveillance Radar - 1988 Doppler (WSR-88D) level II (base) data from 120 National Weather Service (NWS), 12 Federal Aviation Administration (FAA) and 26 Department of Defense (DOD) sites on 8mm tape or online. The NCDC robotic mass storage system warehouses approximately 900 terabytes of Level II data and grows annually at a rate of 60 terabytes a year (120 terabytes with offsite backup). Over the next several years enhanced radar technologies, such as dual polarization, will be implemented and growth of the radar digital archive may increase by a factor of 26. The NCDC is partnering with the Radar Operations Center, the National Severe Storms Lab, the University of Oklahoma and Unidata on the Collaborative Radar Acquisition Field Test (CRAFT) project. CRAFT started as a grass roots effort to transmit level II data electronically, from 6 WSR-88D sites directly to the NCDC robotic mass storage system. To date, 59 of the 158 WSR-88D sites are transmitting level II data in real time to NCDC. NCDC has truly become a \"one stop shop\" for WSR-88D radar users. Direct digital access to level II radar inventories, data, and visualization software are available, at no cost to the user, via the NCDC radar resources web page. (http://lwf.ncdc.noaa.gov/oa/radarresources.html) Large data volume (in the past, took weeks to months to disseminate) are accessible in minutes to hours.","PeriodicalId":285161,"journal":{"name":"20th IEEE/11th NASA Goddard Conference on Mass Storage Systems and Technologies, 2003. (MSST 2003). Proceedings.","volume":"3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"1900-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123779956","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}