大型数据集的广义压缩和压缩搜索

Morgan E. Prior, Thomas Howard III, Emily Light, Najib Ishaq, Noah M. Daniels
{"title":"大型数据集的广义压缩和压缩搜索","authors":"Morgan E. Prior, Thomas Howard III, Emily Light, Najib Ishaq, Noah M. Daniels","doi":"arxiv-2409.12161","DOIUrl":null,"url":null,"abstract":"The Big Data explosion has necessitated the development of search algorithms\nthat scale sub-linearly in time and memory. While compression algorithms and search algorithms do exist independently,\nfew algorithms offer both, and those which do are domain-specific. We present panCAKES, a novel approach to compressive search, i.e., a way to\nperform $k$-NN and $\\rho$-NN search on compressed data while only decompressing\na small, relevant, portion of the data. panCAKES assumes the manifold hypothesis and leverages the low-dimensional\nstructure of the data to compress and search it efficiently. panCAKES is generic over any distance function for which the distance between\ntwo points is proportional to the memory cost of storing an encoding of one in\nterms of the other. This property holds for many widely-used distance functions, e.g. string edit\ndistances (Levenshtein, Needleman-Wunsch, etc.) and set dissimilarity measures\n(Jaccard, Dice, etc.). We benchmark panCAKES on a variety of datasets, including genomic, proteomic,\nand set data. We compare compression ratios to gzip, and search performance between the\ncompressed and uncompressed versions of the same dataset. panCAKES achieves compression ratios close to those of gzip, while offering\nsub-linear time performance for $k$-NN and $\\rho$-NN search. We conclude that panCAKES is an efficient, general-purpose algorithm for\nexact compressive search on large datasets that obey the manifold hypothesis. We provide an open-source implementation of panCAKES in the Rust programming\nlanguage.","PeriodicalId":501281,"journal":{"name":"arXiv - CS - Information Retrieval","volume":"15 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Generalized compression and compressive search of large datasets\",\"authors\":\"Morgan E. Prior, Thomas Howard III, Emily Light, Najib Ishaq, Noah M. Daniels\",\"doi\":\"arxiv-2409.12161\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"The Big Data explosion has necessitated the development of search algorithms\\nthat scale sub-linearly in time and memory. While compression algorithms and search algorithms do exist independently,\\nfew algorithms offer both, and those which do are domain-specific. We present panCAKES, a novel approach to compressive search, i.e., a way to\\nperform $k$-NN and $\\\\rho$-NN search on compressed data while only decompressing\\na small, relevant, portion of the data. panCAKES assumes the manifold hypothesis and leverages the low-dimensional\\nstructure of the data to compress and search it efficiently. panCAKES is generic over any distance function for which the distance between\\ntwo points is proportional to the memory cost of storing an encoding of one in\\nterms of the other. This property holds for many widely-used distance functions, e.g. string edit\\ndistances (Levenshtein, Needleman-Wunsch, etc.) and set dissimilarity measures\\n(Jaccard, Dice, etc.). We benchmark panCAKES on a variety of datasets, including genomic, proteomic,\\nand set data. We compare compression ratios to gzip, and search performance between the\\ncompressed and uncompressed versions of the same dataset. panCAKES achieves compression ratios close to those of gzip, while offering\\nsub-linear time performance for $k$-NN and $\\\\rho$-NN search. We conclude that panCAKES is an efficient, general-purpose algorithm for\\nexact compressive search on large datasets that obey the manifold hypothesis. We provide an open-source implementation of panCAKES in the Rust programming\\nlanguage.\",\"PeriodicalId\":501281,\"journal\":{\"name\":\"arXiv - CS - Information Retrieval\",\"volume\":\"15 1\",\"pages\":\"\"},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-09-18\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"arXiv - CS - Information Retrieval\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/arxiv-2409.12161\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Information Retrieval","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.12161","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

摘要

大数据爆炸要求开发在时间和内存上呈亚线性扩展的搜索算法。虽然压缩算法和搜索算法各自独立存在,但很少有算法能同时提供压缩算法和搜索算法,而且提供压缩算法和搜索算法的算法都是针对特定领域的。PanCAKES 假设流形假设,并利用数据的低维结构来高效地压缩和搜索数据。PanCAKES 通用于任何距离函数,两点之间的距离与存储一个点与另一个点之间的编码的内存成本成正比。这一特性适用于许多广泛使用的距离函数,例如字符串编辑距离(Levenshtein、Needleman-Wunsch 等)和集合不相似度量(Jaccard、Dice 等)。我们在各种数据集(包括基因组、蛋白质组和集合数据)上对 panCAKES 进行了基准测试。我们比较了与 gzip 的压缩率,以及同一数据集压缩版本和未压缩版本的搜索性能。panCAKES 实现了接近 gzip 的压缩率,同时为 $k$-NN 和 $\rho$-NN 搜索提供了近线性时间性能。我们的结论是,panCAKES 是一种高效的通用算法,可以在符合流形假设的大型数据集上进行压缩搜索。我们用 Rust 编程语言提供了 panCAKES 的开源实现。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
Generalized compression and compressive search of large datasets
The Big Data explosion has necessitated the development of search algorithms that scale sub-linearly in time and memory. While compression algorithms and search algorithms do exist independently, few algorithms offer both, and those which do are domain-specific. We present panCAKES, a novel approach to compressive search, i.e., a way to perform $k$-NN and $\rho$-NN search on compressed data while only decompressing a small, relevant, portion of the data. panCAKES assumes the manifold hypothesis and leverages the low-dimensional structure of the data to compress and search it efficiently. panCAKES is generic over any distance function for which the distance between two points is proportional to the memory cost of storing an encoding of one in terms of the other. This property holds for many widely-used distance functions, e.g. string edit distances (Levenshtein, Needleman-Wunsch, etc.) and set dissimilarity measures (Jaccard, Dice, etc.). We benchmark panCAKES on a variety of datasets, including genomic, proteomic, and set data. We compare compression ratios to gzip, and search performance between the compressed and uncompressed versions of the same dataset. panCAKES achieves compression ratios close to those of gzip, while offering sub-linear time performance for $k$-NN and $\rho$-NN search. We conclude that panCAKES is an efficient, general-purpose algorithm for exact compressive search on large datasets that obey the manifold hypothesis. We provide an open-source implementation of panCAKES in the Rust programming language.
求助全文
通过发布文献求助,成功后即可免费获取论文全文。 去求助
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信