A Reference Method for Performance Evaluation in Big Data Architectures

Abstract

This paper presents a reference method for performance evaluation in Big Data architectures, called by Improvement Method for Big Data Architectures (IMBDA) aiming to increase the performance, and consequently raising the quality of service provided. The method will contribute to small businesses and startups that have limited financial re-sources (impossible to invest in market solutions). The proposed approach considers the relationship of the processes in a data processing flow to find possible bottlenecks and optimization points. To this end, IMBDA collects system logs to compose functional metrics (e.g., processing time) and non-functional metrics (e.g., CPU and memory utilization, and other cloud computing infrastructure resources). The system stores these metrics in an external data analysis tool that investigates the correlation of performance between processes. The reference method applies to the architecture of a Big Data application, which provides solutions in fleet logistics. With the use of IMBDA, it was possible to identify performance bottlenecks, allowing the reconfiguration of the architecture to increase service quality at the lowest possible cost.