Secure and multi-tenant Hadoop cluster - an experience

Abstract

Data Analytics and Data Discovery are the most important facets in today's Business Domain where customer centric business decisions are the key. With ever increasing rate of data captivation, curation, management and requirement of data analytics, Hadoop has accounted itself as a major player in providing the data analytics and data processing backbone for any Organization that deals with ever increasing nuances of data management and processing. With every organizational setup of Hadoop clusters, we find it an ever increasing challenge to setup, manage and operate multiple Hadoop clusters, for managing different projects or managing different Tenants (clients). This results in a higher client onboarding time on Hadoop, cost of project ownership and effort to setup and manage separate clusters for separate projects/clients/tenants. However with the current trend of data security, companies are apprehensive of building a single large cluster and onboarding multiple clients on same common Hadoop cluster. This paper demonstrates how to set up a multi-tenant cluster which is big in size, scalable enough and has short client onboarding time without any client having access/knowledge/information of any other clients. Security features are also implemented on this multi-tenant cluster for authentication and authorization, so that only right client members have access to their allocated Hadoop resources like RAM, CPU and disk size. This paper also demonstrates how to create a fully functional and operational multi-tenant cluster with security at its core, reduced Cluster Management, higher data & resource security to provide an optimized Hadoop based solution offering in terms of cost and effectiveness.