Data compression and query for large scale sensor data on COTS DBMS

Abstract

Multi-dimensional temporal data set is the common format in sensor network applications to store sampled temporal data. As time goes on, the size of the core tables in the data set may increase to enormous size and the tables become not managable. In order to reduce storage space and allow on-line query, how to trade off data compression effectiveness for on-line query performance is a challenge issue. In this paper, we are concerned with an effective framework for temporal data set that does not scarify on-line query performance and is specifically designed for very large sensor network database. The sampled data are compressed using several candidate approaches including dictionary-base compress and lossless vector quantization. In the mean time, on-line queries are conducted without decompressing the compressed data set so as to enhance the query performance. Experiments are conducted on a power meter database and sonoma database to evaluate the proposed methodologies in terms of data compression rate and data query speed. The results show that the compression rate ranges from 70% for numerical data to 20% for character data. In the mean time, the increased overhead for online query is limited up to 2%.