Limits of data stream compression under the L∞ norm

Abstract

As vast environmental monitoring projects continue to proliferate, the problem of efficient data representation becomes more and more significant. We tackle the fundamental question of what is the limit of lossy compression of a data stream under the L∞ norm. We describe a method to compute a conservative estimate of the entropy of a sequence of non-independent random variables underlying a data stream. We find experimentally that the conservative limit estimation lies as low as 1/5 of the best performing practical representation methods.