Software engineering issues in big data application development

Abstract

Big Data has become one of the most important concepts that is being studied in Computer/Software Engineering. The data produced in recent years have increased rapidly and exponentially, necessitating the solution of major problems such as the collection, processing and storage of huge volume of data. Big Data Frameworks are developed specifically to solve these problems that facilitates application developers by providing opportunities to collect, process, manage, monitor and analyze these data. A few examples of these frameworks are Hadoop, Spark, Storm, and Flink, which are developed by Software Engineers as open source projects. Although the challenges raised from coordination of IT resources such as huge amounts of computation power, storage area, memory, and network bandwidth in a distributed manner solved by these frameworks, there still remains many Software Engineering problems in application development phase, even if they based on these frameworks. High scalability, fault tolerance, flexibility, reliability and testability can be listed as the main issues need to be carefully considered in terms of Software Engineering. In this paper, we first clarify the terms Framework-Application, and then the overview information about Big Data and related frameworks are given before emphasizing the problems arising in terms of Software Engineering. Nevertheless, we tried to provide guidance to the people who would develop software for Big Data and tried to give the further research guidance.