Synergizing Hybrid Agile-Scrum and CRISP-DM Approaches in Data Science Project Management

Delivering a data science project encompasses several hurdles that need to be addressed. As the project matures, the business requirements may change over time. In addition, uncertainties associated with data integrity, quantity, and quality can have an impact on the success of the project. The effectiveness of advanced analytics and algorithms that is changing depending on the complexity of the project and ambiguities in project values potentially will lead to adverse effects on deliverables and task prioritization. Agile-scrum framework enables projects with time-boxed iterations (sprints). It also introduces delivery through increments (MVPs) that assist in reaching the overall aim or vision of the product. However, backlog prioritization and sequence of tasks is not bounded by any criteria and depends fully on product owner’s understanding of product goal and value. On the other hand, CRISP-DM is a solid place to start for advising developers on the steps and tasks needed to build a data science product. It enables exploratory and discovery work through iterations to satisfy the requirements of the data science project. However, the lack of time element within the process might cause infinite iterative cycles and delay delivery to customers. At Petronas, we have integrated a hybrid strategy that envelops the CRISP-DM process within defined time-limited sprints. The process flow from CRISP-DM can help to plan which tasks to be assigned in which sprint. Properly assigned scrum team roles will ensure proper establishment of scrum. Furthermore, conducting scrum events will enable effective and productive customers engagement. Periodic inspection of scrum artifacts will also ensure alignment with product goals. This hybrid approach demonstrates how the change in requirements can be strategically addressed by utilizing the Agile-Scrum CRISP-DM methodology while ensuring that the product goal is achieved. It also highlights how Agile-Scrum ensures successful delivery of product, maximizing product value, and customer satisfaction, while CRISP-DM can guide us in planning for data science project sprints.