Deconstructing the Lambda Architecture: An Experience Report

Abstract

In this paper we propose a new architecture for the development of big data projects which combine real time and batch processing. The starting point was the Lambda architecture, but several important limitations were detected when applying it to a real big data project. To solve all these issues and to be able to develop the project in a more satisfactory manner, the Lambda architecture was evolved, and as a result we have created a new and more flexible architecture. With this new architecture we were able to complete our project successfully, optimizing hardware usage, using a smaller development team and making the final result easier to maintain. Based in our experience, this new architecture, called Phi, seems to be generic enough to be widely applied to big data projects. This architecture, though more specific than Lambda, could improve and make easier the development and evolution of such projects.