A Meta Learning Approach for Automating Model Selection in Big Data Environments using Microservice and Container Virtualization Technologies

For a given specific machine learning task, very often several machine learning algorithms and their right configurations are tested in a trial-and-error approach, until an adequate solution is found. This wastes human resources for constructing multiple models, requires a data analytics expert and is time-consuming, since a variety of learning algorithms are proposed in literature and the non-expert users do not know which one to use in order to obtain good performance results. Meta learning addresses these problems and supports non-expert users by recommending a promising learning algorithm based on meta features computed from a given dataset. In the present paper, a new generic microservice-based framework for realizing the concept of meta learning in Big Data environments is introduced. This framework makes use of a powerful Big Data software stack, container visualization, modern web technologies and a microservice architecture for a fully manageable and highly scalable solution. In this demonstration and for evaluation purpose, time series model selection is taken into account. The performance and usability of the new framework is evaluated on state-of-the-art machine learning algorithms for time series forecasting: it is shown that the proposed microservice-based meta learning framework introduces an excellent performance in assigning the adequate forecasting model for the chosen time series datasets. Moreover, the recommendation of the most appropriate forecasting model results in a well acceptable low overhead demonstrating that the framework can provide an efficient approach to solve the problem of model selection in context of Big Data.