Machine Learning System Architectural Pattern for Improving Operational Stability

Recently, machine learning systems with inference engines have been widely used for a variety of purposes (such as prediction and classification) in our society. While it is quite important to keep the services provided by these machine learning systems stable, maintaining stability can be difficult given the nature of machine learning systems whose behaviors can be determined by program codes and input data. Therefore, quick troubleshooting (problem localization, rollback, etc.) is necessary. However, common machine learning systems with three-layer architectural patterns complicate the troubleshooting process because of their tightly coupled functions (e.g., business logic coded from design and inference engine derived from data). To solve the problem, we propose a novel architectural pattern for machine learning systems in which components for business logic and components for machine learning are separated. This architectural pattern helps operators break down the failures into a business logic part and a ML-specific part, and they can rollback the inference engine independent of the business logic when the inference engine has some problems. Through a practical case study scenario, we will show how our architectural pattern can make troubleshooting easier than common three-layer architecture.