Revealing Challenges within the Application of Machine Learning Services - A Delphi Study

Over the past years, Machine Learning has been applied to an increasing number of problems across numerous industries. However, the steady rise in the application of Machine Learning has not come without challenges since companies often lack the expertise or infrastructure to build their own Machine Learning systems. These challenges led to the emergence of a new paradigm, called Machine Learning as a Service. Scientific literature has mainly analyzed this topic in the context of platform solutions that provide ready-to-use environments for companies. We recently have developed a platform-independent approach and labeled it Machine Learning Services. The aim of the present study is to identify and evaluate challenges and opportunities in the application of Machine Learning Services. To do so, we conducted a Delphi Study with a panel of machine learning experts. The study consisted of three rounds and was structured according to the five steps of the Data Science Lifecycle. A variety of challenges from the areas “Communication”, “Environment”, “Approach”, “Data”, “Retraining, Testing, Monitoring and Updating”, “Model Training and Evaluation” were identified. Subsequently, the challenges revealed by the Delphi Study were compared with previous work on Machine Learning as a Service, which resulted from a structured literature review. The identified areas serve as possible future research fields and give further implications for practice. Alleviating communication issues and assessing the business IT infrastructure prior to the machine learning project are among the key findings of our study.