An IoT-oriented real-time storage mechanism for massive small files based on Swift

Abstract

In the internet of things (IoT), large amounts of small files are generated from various structure sensors in cloud storage platforms. Real-time storage of massive small files will put great pressures to the traditional file system. By studying the real-time storage strategy of massive small files based on the object storage architecture Swift, we proved a unique aggregation storage strategy, called sequential data aggregation strategy (SDAS), for storage of small files. We designed a two-level index structure to improve writing rate by transferring randomly write to sequentially write. To improve overall data access efficiency and solve the performance bottleneck of proxy node, we utilise a file's potential relevance of timing to prefetch related files that are merged in other blocks. Simulation results show that relative to the original system, SDAS has shorter response time of writing operation, lower cost of index maintenance cost, more balanced node load and 30% reduction in disk overhead.