IoT-Dedup: Device Relationship-Based IoT Data Deduplication Scheme

The cyclical and continuous working characteristics of Internet of Things (IoT) devices make a large amount of the same or similar data, which can significantly consume storage space. To solve this problem, various secure data deduplication schemes have been proposed. However, existing deduplication schemes only perform deduplication based on data similarity, ignoring the internal connection among devices, making the existing schemes not directly applicable to parallel and distributed scenarios like IoT. Furthermore, since secure data deduplication leads to multiple users sharing same encryption key, which may lead to security issues. To this end, we propose a device relationship-based IoT data deduplication scheme that fully considers the IoT data characteristics and devices internal connections. Specifically, we propose a device relationship prediction approach, which can obtain device collaborative relationships by clustering the topology of their communication graph, and classifies the data types based on device relationships to achieve data deduplication with different security levels. Then, we design a similarity-preserving encryption algorithm, so that the security level of encryption key is determined by the data type, ensuring the security of the deduplicated data. In addition, two different data deduplication methods, identical deduplication and similar deduplication, have been designed to meet the privacy requirement of different data types, improving the efficiency of deduplication while ensuring data privacy as much as possible. We evaluate the performance of our scheme using five real datasets, and the results show that our scheme has favorable results in terms of both deduplication performance and computational cost.