A multivocal literature review on the benefits and limitations of industry-leading AutoML tools

Abstract

Context:

Rapid advancements in Artificial Intelligence (AI) and Machine Learning (ML) are revolutionizing software engineering in every application domain, driving unprecedented transformations and fostering innovation. However, despite these advances, several organizations are experiencing friction in the adoption of ML-based technologies, mainly due to the current shortage of ML professionals. In this context, Automated Machine Learning (AutoML) techniques have been presented as a promising solution to democratize ML adoption, even in the absence of specialized people.

Objective:

Our research aims to provide an overview of the evidence on the benefits and limitations of AutoML tools being adopted in industry.

Methods:

We conducted a Multivocal Literature Review, which allowed us to identify 54 sources from the academic literature and 108 sources from the grey literature reporting on AutoML benefits and limitations. We extracted explicitly reported benefits and limitations from the papers and applied the thematic analysis method for synthesis.

Results:

In general, we identified 18 reported benefits and 25 limitations. Concerning the benefits, we highlight that AutoML tools can help streamline the core steps of ML workflows, namely data preparation, feature engineering, model construction, and hyperparameter tuning—with concrete benefits on model performance, efficiency, and scalability. In addition, AutoML empowers both novice and experienced data scientists, promoting ML accessibility. However, we highlight several limitations that may represent obstacles to the widespread adoption of AutoML. For instance, AutoML tools may introduce barriers to transparency and interoperability, exhibit limited flexibility for complex scenarios, and offer inconsistent coverage of the ML workflow.

Conclusion:

The effectiveness of AutoML in facilitating the adoption of machine learning by users may vary depending on the specific tool and the context in which it is used. Today, AutoML tools are used to increase human expertise rather than replace it and, as such, require skilled users.