A Study on the Performance and Scalability of Apache Flink Over Hadoop MapReduce

Int. J. Fog Comput. Pub Date : 2019-01-01 DOI:10.4018/IJFC.2019010103

Pankaj Lathar, K. Srinivasa

{"title":"A Study on the Performance and Scalability of Apache Flink Over Hadoop MapReduce","authors":"Pankaj Lathar, K. Srinivasa","doi":"10.4018/IJFC.2019010103","DOIUrl":null,"url":null,"abstract":"With the advancements in science and technology, data is being generated at a staggering rate. The raw data generated is generally of high value and may conceal important information with the potential to solve several real-world problems. In order to extract this information, the raw data available must be processed and analysed efficiently. It has however been observed, that such raw data is generated at a rate faster than it can be processed by traditional methods. This has led to the emergence of the popular parallel processing programming model – MapReduce. In this study, the authors perform a comparative analysis of two popular data processing engines – Apache Flink and Hadoop MapReduce. The analysis is based on the parameters of scalability, reliability and efficiency. The results reveal that Flink unambiguously outperformance Hadoop's MapReduce. Flink's edge over MapReduce can be attributed to following features – Active Memory Management, Dataflow Pipelining and an Inline Optimizer. It can be concluded that as the complexity and magnitude of real time raw data is continuously increasing, it is essential to explore newer platforms that are adequately and efficiently capable of processing such data.","PeriodicalId":218786,"journal":{"name":"Int. J. Fog Comput.","volume":"26 1","pages":"0"},"PeriodicalIF":0.0000,"publicationDate":"2019-01-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"2","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"Int. J. Fog Comput.","FirstCategoryId":"1085","ListUrlMain":"https://doi.org/10.4018/IJFC.2019010103","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}

引用次数: 2

Abstract

With the advancements in science and technology, data is being generated at a staggering rate. The raw data generated is generally of high value and may conceal important information with the potential to solve several real-world problems. In order to extract this information, the raw data available must be processed and analysed efficiently. It has however been observed, that such raw data is generated at a rate faster than it can be processed by traditional methods. This has led to the emergence of the popular parallel processing programming model – MapReduce. In this study, the authors perform a comparative analysis of two popular data processing engines – Apache Flink and Hadoop MapReduce. The analysis is based on the parameters of scalability, reliability and efficiency. The results reveal that Flink unambiguously outperformance Hadoop's MapReduce. Flink's edge over MapReduce can be attributed to following features – Active Memory Management, Dataflow Pipelining and an Inline Optimizer. It can be concluded that as the complexity and magnitude of real time raw data is continuously increasing, it is essential to explore newer platforms that are adequately and efficiently capable of processing such data.

查看原文本刊更多论文

基于Hadoop MapReduce的Apache Flink性能与可扩展性研究

随着科学技术的进步，数据正以惊人的速度产生。生成的原始数据通常具有很高的价值，并且可能隐藏有可能解决几个实际问题的重要信息。为了提取这些信息，必须对可用的原始数据进行有效的处理和分析。然而，人们观察到，这些原始数据的生成速度比传统方法处理的速度要快。这导致了流行的并行处理编程模型MapReduce的出现。在这项研究中，作者对两种流行的数据处理引擎——Apache Flink和Hadoop MapReduce进行了比较分析。该分析基于可扩展性、可靠性和效率等参数。结果显示，Flink的性能明显优于Hadoop的MapReduce。Flink相对于MapReduce的优势可以归因于以下特性——主动内存管理、数据流流水线和内联优化器。可以得出的结论是，随着实时原始数据的复杂性和规模不断增加，探索能够充分有效地处理此类数据的新平台至关重要。

本文章由计算机程序翻译，如有差异，请以英文原文为准。

求助全文

约1分钟内获得全文求助全文

来源期刊

Int. J. Fog Comput.

自引率

0.00%

发文量