Morgan E. Prior, Thomas Howard III, Emily Light, Najib Ishaq, Noah M. Daniels
{"title":"Generalized compression and compressive search of large datasets","authors":"Morgan E. Prior, Thomas Howard III, Emily Light, Najib Ishaq, Noah M. Daniels","doi":"arxiv-2409.12161","DOIUrl":null,"url":null,"abstract":"The Big Data explosion has necessitated the development of search algorithms\nthat scale sub-linearly in time and memory. While compression algorithms and search algorithms do exist independently,\nfew algorithms offer both, and those which do are domain-specific. We present panCAKES, a novel approach to compressive search, i.e., a way to\nperform $k$-NN and $\\rho$-NN search on compressed data while only decompressing\na small, relevant, portion of the data. panCAKES assumes the manifold hypothesis and leverages the low-dimensional\nstructure of the data to compress and search it efficiently. panCAKES is generic over any distance function for which the distance between\ntwo points is proportional to the memory cost of storing an encoding of one in\nterms of the other. This property holds for many widely-used distance functions, e.g. string edit\ndistances (Levenshtein, Needleman-Wunsch, etc.) and set dissimilarity measures\n(Jaccard, Dice, etc.). We benchmark panCAKES on a variety of datasets, including genomic, proteomic,\nand set data. We compare compression ratios to gzip, and search performance between the\ncompressed and uncompressed versions of the same dataset. panCAKES achieves compression ratios close to those of gzip, while offering\nsub-linear time performance for $k$-NN and $\\rho$-NN search. We conclude that panCAKES is an efficient, general-purpose algorithm for\nexact compressive search on large datasets that obey the manifold hypothesis. We provide an open-source implementation of panCAKES in the Rust programming\nlanguage.","PeriodicalId":501281,"journal":{"name":"arXiv - CS - Information Retrieval","volume":"15 1","pages":""},"PeriodicalIF":0.0000,"publicationDate":"2024-09-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"arXiv - CS - Information Retrieval","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/arxiv-2409.12161","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0
Abstract
The Big Data explosion has necessitated the development of search algorithms
that scale sub-linearly in time and memory. While compression algorithms and search algorithms do exist independently,
few algorithms offer both, and those which do are domain-specific. We present panCAKES, a novel approach to compressive search, i.e., a way to
perform $k$-NN and $\rho$-NN search on compressed data while only decompressing
a small, relevant, portion of the data. panCAKES assumes the manifold hypothesis and leverages the low-dimensional
structure of the data to compress and search it efficiently. panCAKES is generic over any distance function for which the distance between
two points is proportional to the memory cost of storing an encoding of one in
terms of the other. This property holds for many widely-used distance functions, e.g. string edit
distances (Levenshtein, Needleman-Wunsch, etc.) and set dissimilarity measures
(Jaccard, Dice, etc.). We benchmark panCAKES on a variety of datasets, including genomic, proteomic,
and set data. We compare compression ratios to gzip, and search performance between the
compressed and uncompressed versions of the same dataset. panCAKES achieves compression ratios close to those of gzip, while offering
sub-linear time performance for $k$-NN and $\rho$-NN search. We conclude that panCAKES is an efficient, general-purpose algorithm for
exact compressive search on large datasets that obey the manifold hypothesis. We provide an open-source implementation of panCAKES in the Rust programming
language.