Performance Optimization of In-Memory File System in Distributed Storage System

Abstract

Hadoop as an open source framework for dealing with Big Data can be processed to calculate large amounts of data in parallel, which has attracted more and more attention in academia and industry. This paper analyzes the methods of In-Memory File System using HDFS Lazy Persist strategy and Alluxio to upgrade system I/O efficiency. Besides, in order to avoid the problem that Lazy Persist strategy needs to be triggered manually each time, we propose HDFS Lazy Persist strategy automatic trigger mechanism based on the statistics of data access information.