Yu Chen, Jun Zhang, Zhicheng Wang, Gejian Liao, Shu Liu, Hai Tan, Guowei Yang, Ying Fang, Shuai Wang, Zhaoqun Sun
{"title":"一种基于Hadoop的小文件快速读少存储算法","authors":"Yu Chen, Jun Zhang, Zhicheng Wang, Gejian Liao, Shu Liu, Hai Tan, Guowei Yang, Ying Fang, Shuai Wang, Zhaoqun Sun","doi":"10.1109/ICCEAI52939.2021.00040","DOIUrl":null,"url":null,"abstract":"Massive small files access is the main challenge for the Hadoop Distributed File System. To solve these problems, we present a new Algorithm of archive file, A Faster Read and Less Storage Algorithm for Small Files on Hadoop. A new logical file name is used to identify the file which generated by the pair in the name node. Our experiments show that the algorithm is around 76.6% faster than original HDFS in the time of file storing, and around 31.9.6% faster than original HDFS in the time of file reading, around 73.9% less than original HDFS in the memory consumption of namenode.","PeriodicalId":331409,"journal":{"name":"2021 International Conference on Computer Engineering and Artificial Intelligence (ICCEAI)","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2021-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"1","resultStr":"{\"title\":\"A Faster Read and Less Storage Algorithm for Small Files on Hadoop\",\"authors\":\"Yu Chen, Jun Zhang, Zhicheng Wang, Gejian Liao, Shu Liu, Hai Tan, Guowei Yang, Ying Fang, Shuai Wang, Zhaoqun Sun\",\"doi\":\"10.1109/ICCEAI52939.2021.00040\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Massive small files access is the main challenge for the Hadoop Distributed File System. To solve these problems, we present a new Algorithm of archive file, A Faster Read and Less Storage Algorithm for Small Files on Hadoop. A new logical file name is used to identify the file which generated by the pair in the name node. Our experiments show that the algorithm is around 76.6% faster than original HDFS in the time of file storing, and around 31.9.6% faster than original HDFS in the time of file reading, around 73.9% less than original HDFS in the memory consumption of namenode.\",\"PeriodicalId\":331409,\"journal\":{\"name\":\"2021 International Conference on Computer Engineering and Artificial Intelligence (ICCEAI)\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2021-08-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"1\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"2021 International Conference on Computer Engineering and Artificial Intelligence (ICCEAI)\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.1109/ICCEAI52939.2021.00040\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"2021 International Conference on Computer Engineering and Artificial Intelligence (ICCEAI)","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.1109/ICCEAI52939.2021.00040","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
A Faster Read and Less Storage Algorithm for Small Files on Hadoop
Massive small files access is the main challenge for the Hadoop Distributed File System. To solve these problems, we present a new Algorithm of archive file, A Faster Read and Less Storage Algorithm for Small Files on Hadoop. A new logical file name is used to identify the file which generated by the pair in the name node. Our experiments show that the algorithm is around 76.6% faster than original HDFS in the time of file storing, and around 31.9.6% faster than original HDFS in the time of file reading, around 73.9% less than original HDFS in the memory consumption of namenode.