Effort Estimation Method for Extract Transfer Load (ETL) Big Data Projects

Abstract

Effort estimation is a critical stage in the life cycle of a project. Effort underestimation may lead to product quality reduction and trust issues. This also applies to Extract Transfer Load (ETL) Big Data projects. This study aims to propose an effort estimation method related to ETL Big Data projects. A literature review was performed to analyze the key aspects affecting effort estimation for ETL Big Data projects. The output of the review was used to carry out a survey with professionals involved in such projects to identify factors and methodologies influential to ETL Big Data effort estimation. The findings from the literature and the survey results were used to propose a method that can be used for effort estimation for ETL Big Data projects. The proposed solution is to implement the current estimation method of an organization and the Constructive Cost Model (COCOMO) II method which are aligned with factors related to ETL Big Data projects. The proposed solution was evaluated using a real case study of a multinational company. The COCOMO II method was found to be slightly better than the manual estimation method currently used. Although further evaluation is required with different types of ETL Big Data projects, based on these initial findings, we can say that COCOMO II can be used for such projects.