aws-s3-integrity-check: an open-source bash tool to verify the integrity of a dataset stored on Amazon S3.

GigaByte (Hong Kong, China) Pub Date : 2023-08-23 eCollection Date: 2023-01-01 DOI:10.46471/gigabyte.87

Sonia García-Ruiz, Regina Hertfelder Reynolds, Melissa Grant-Peters, Emil Karl Gustavsson, Aine Fairbrother-Browne, Zhongbo Chen, Jonathan William Brenton, Mina Ryten

{"title":"aws-s3-integrity-check: an open-source bash tool to verify the integrity of a dataset stored on Amazon S3.","authors":"Sonia García-Ruiz, Regina Hertfelder Reynolds, Melissa Grant-Peters, Emil Karl Gustavsson, Aine Fairbrother-Browne, Zhongbo Chen, Jonathan William Brenton, Mina Ryten","doi":"10.46471/gigabyte.87","DOIUrl":null,"url":null,"abstract":"Amazon Simple Storage Service (Amazon S3) is a widely used platform for storing large biomedical datasets. Unintended data alterations can occur during data writing and transmission, altering the original content and generating unexpected results. However, no open-source and easy-to-use tool exists to verify end-to-end data integrity. Here, we present aws-s3-integrity-check, a user-friendly, lightweight, and reliable bash tool to verify the integrity of a dataset stored in an Amazon S3 bucket. Using this tool, we only needed ∼114 min to verify the integrity of 1,045 records ranging between 5 bytes and 10 gigabytes and occupying ∼935 gigabytes of the Amazon S3 cloud. Our aws-s3-integrity-check tool also provides file-by-file on-screen and log-file-based information about the status of each integrity check. To our knowledge, this tool is the only open-source one that allows verifying the integrity of a dataset uploaded to the Amazon S3 Storage quickly, reliably, and efficiently. The tool is freely available for download and use at https://github.com/SoniaRuiz/aws-s3-integrity-check and https://hub.docker.com/r/soniaruiz/aws-s3-integrity-check.","PeriodicalId":73157,"journal":{"name":"GigaByte (Hong Kong, China)","volume":"2023 ","pages":"gigabyte87"},"PeriodicalIF":0.0000,"publicationDate":"2023-08-23","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10448181/pdf/","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"GigaByte (Hong Kong, China)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.46471/gigabyte.87","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2023/1/1 0:00:00","PubModel":"eCollection","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 0

Abstract

Amazon Simple Storage Service (Amazon S3) is a widely used platform for storing large biomedical datasets. Unintended data alterations can occur during data writing and transmission, altering the original content and generating unexpected results. However, no open-source and easy-to-use tool exists to verify end-to-end data integrity. Here, we present aws-s3-integrity-check, a user-friendly, lightweight, and reliable bash tool to verify the integrity of a dataset stored in an Amazon S3 bucket. Using this tool, we only needed ∼114 min to verify the integrity of 1,045 records ranging between 5 bytes and 10 gigabytes and occupying ∼935 gigabytes of the Amazon S3 cloud. Our aws-s3-integrity-check tool also provides file-by-file on-screen and log-file-based information about the status of each integrity check. To our knowledge, this tool is the only open-source one that allows verifying the integrity of a dataset uploaded to the Amazon S3 Storage quickly, reliably, and efficiently. The tool is freely available for download and use at https://github.com/SoniaRuiz/aws-s3-integrity-check and https://hub.docker.com/r/soniaruiz/aws-s3-integrity-check.

Abstract Image

查看原文本刊更多论文

aws-s3-integrity-check：一款开源 bash 工具，用于验证存储在亚马逊 S3 上的数据集的完整性。

亚马逊简单存储服务（Amazon S3）是一个广泛用于存储大型生物医学数据集的平台。在数据写入和传输过程中，可能会发生意外的数据更改，从而改变原始内容并产生意想不到的结果。然而，目前还没有开源且易于使用的工具来验证端到端的数据完整性。在此，我们介绍 aws-s3-integrity-check，这是一款用户友好、轻量级且可靠的 bash 工具，用于验证亚马逊 S3 存储桶中存储的数据集的完整性。使用该工具，我们只用了 114 分钟就验证了亚马逊 S3 云中 1,045 条记录的完整性，这些记录的大小从 5 字节到 10 千兆字节不等，占用了 935 千兆字节的空间。我们的 aws-s3-integrity-check 工具还在屏幕上提供了逐个文件的信息，并在日志文件中提供了每次完整性检查的状态信息。据我们所知，该工具是唯一一款可以快速、可靠、高效地验证上传到亚马逊 S3 存储的数据集完整性的开源工具。该工具可在 https://github.com/SoniaRuiz/aws-s3-integrity-check 和 https://hub.docker.com/r/soniaruiz/aws-s3-integrity-check 免费下载和使用。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊