Rabie Mahdaoui, Manar Sais, J. Abouchabaka, N. Rafalia
{"title":"利用多代理系统提高 Hadoop 分布式存储效率","authors":"Rabie Mahdaoui, Manar Sais, J. Abouchabaka, N. Rafalia","doi":"10.11591/ijeecs.v34.i3.pp1814-1822","DOIUrl":null,"url":null,"abstract":"Distributed storage systems play a pivotal role in modern data-intensive applications, with Hadoop distributed file system (HDFS) being a prominent example. However, optimizing the efficiency of such systems remains a complex challenge. This research paper presents a novel approach to enhance the efficiency of distributed storage by leveraging multi-agent systems (MAS). Our research is centered on enhancing the efficiency of the HDFS by incorporating intelligent agents that can dynamically assign storage tasks to nodes based on their performance characteristics. Utilizing a decentralized decision-making framework, the suggested approach based on MAS considers the real-time performance of nodes and allocates storage tasks adaptively. This strategy aims to alleviate performance bottlenecks and minimize data transfer latency. Through extensive experimental evaluation, we demonstrate the effectiveness of our approach in improving HDFS performance in terms of data storage, retrieval, and overall system efficiency. The results reveal significant reductions in job execution times and enhanced resource utilization, there by offering a promising avenue for enhancing the efficiency of distributed storage systems.","PeriodicalId":13480,"journal":{"name":"Indonesian Journal of Electrical Engineering and Computer Science","volume":null,"pages":null},"PeriodicalIF":0.0000,"publicationDate":"2024-06-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":"{\"title\":\"Enhancing Hadoop distributed storage efficiency using multi-agent systems\",\"authors\":\"Rabie Mahdaoui, Manar Sais, J. Abouchabaka, N. Rafalia\",\"doi\":\"10.11591/ijeecs.v34.i3.pp1814-1822\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"Distributed storage systems play a pivotal role in modern data-intensive applications, with Hadoop distributed file system (HDFS) being a prominent example. However, optimizing the efficiency of such systems remains a complex challenge. This research paper presents a novel approach to enhance the efficiency of distributed storage by leveraging multi-agent systems (MAS). Our research is centered on enhancing the efficiency of the HDFS by incorporating intelligent agents that can dynamically assign storage tasks to nodes based on their performance characteristics. Utilizing a decentralized decision-making framework, the suggested approach based on MAS considers the real-time performance of nodes and allocates storage tasks adaptively. This strategy aims to alleviate performance bottlenecks and minimize data transfer latency. Through extensive experimental evaluation, we demonstrate the effectiveness of our approach in improving HDFS performance in terms of data storage, retrieval, and overall system efficiency. The results reveal significant reductions in job execution times and enhanced resource utilization, there by offering a promising avenue for enhancing the efficiency of distributed storage systems.\",\"PeriodicalId\":13480,\"journal\":{\"name\":\"Indonesian Journal of Electrical Engineering and Computer Science\",\"volume\":null,\"pages\":null},\"PeriodicalIF\":0.0000,\"publicationDate\":\"2024-06-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Indonesian Journal of Electrical Engineering and Computer Science\",\"FirstCategoryId\":\"1085\",\"ListUrlMain\":\"https://doi.org/10.11591/ijeecs.v34.i3.pp1814-1822\",\"RegionNum\":0,\"RegionCategory\":null,\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"\",\"PubModel\":\"\",\"JCR\":\"Q2\",\"JCRName\":\"Mathematics\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Indonesian Journal of Electrical Engineering and Computer Science","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.11591/ijeecs.v34.i3.pp1814-1822","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"Q2","JCRName":"Mathematics","Score":null,"Total":0}
引用次数: 0
摘要
分布式存储系统在现代数据密集型应用中发挥着举足轻重的作用,Hadoop 分布式文件系统(HDFS)就是一个突出的例子。然而,优化此类系统的效率仍然是一项复杂的挑战。本研究论文提出了一种利用多代理系统(MAS)提高分布式存储效率的新方法。我们的研究重点是通过纳入智能代理来提高 HDFS 的效率,这些智能代理可以根据节点的性能特征将存储任务动态分配给节点。基于 MAS 的建议方法利用分散决策框架,考虑节点的实时性能,并自适应地分配存储任务。这一策略旨在缓解性能瓶颈,最大限度地减少数据传输延迟。通过广泛的实验评估,我们证明了我们的方法在提高 HDFS 的数据存储、检索和整体系统效率方面的有效性。结果表明,作业执行时间大幅缩短,资源利用率得到提高,从而为提高分布式存储系统的效率提供了一条大有可为的途径。
Enhancing Hadoop distributed storage efficiency using multi-agent systems
Distributed storage systems play a pivotal role in modern data-intensive applications, with Hadoop distributed file system (HDFS) being a prominent example. However, optimizing the efficiency of such systems remains a complex challenge. This research paper presents a novel approach to enhance the efficiency of distributed storage by leveraging multi-agent systems (MAS). Our research is centered on enhancing the efficiency of the HDFS by incorporating intelligent agents that can dynamically assign storage tasks to nodes based on their performance characteristics. Utilizing a decentralized decision-making framework, the suggested approach based on MAS considers the real-time performance of nodes and allocates storage tasks adaptively. This strategy aims to alleviate performance bottlenecks and minimize data transfer latency. Through extensive experimental evaluation, we demonstrate the effectiveness of our approach in improving HDFS performance in terms of data storage, retrieval, and overall system efficiency. The results reveal significant reductions in job execution times and enhanced resource utilization, there by offering a promising avenue for enhancing the efficiency of distributed storage systems.
期刊介绍:
The aim of Indonesian Journal of Electrical Engineering and Computer Science (formerly TELKOMNIKA Indonesian Journal of Electrical Engineering) is to publish high-quality articles dedicated to all aspects of the latest outstanding developments in the field of electrical engineering. Its scope encompasses the applications of Telecommunication and Information Technology, Applied Computing and Computer, Instrumentation and Control, Electrical (Power), Electronics Engineering and Informatics which covers, but not limited to, the following scope: Signal Processing[...] Electronics[...] Electrical[...] Telecommunication[...] Instrumentation & Control[...] Computing and Informatics[...]