Time-series classification in smart manufacturing systems: An experimental evaluation of state-of-the-art machine learning algorithms

Abstract

Manufacturing is transformed towards smart manufacturing, entering a new data-driven era fueled by digital technologies. The resulting Smart Manufacturing Systems (SMS) gather extensive amounts of diverse data, thanks to the growing number of sensors and rapid advances in sensing technologies. Among the various data types available in SMS settings, time-series data plays a pivotal role. Hence, Time-Series Classification (TSC) emerges as a crucial task in this domain. Over the past decade, researchers have introduced numerous methods for TSC, necessitating not only algorithmic development and analysis but also validation and empirical comparison. This dual approach holds substantial value for practitioners by streamlining choices and revealing insights into models’ strengths and weaknesses. The objective of this study is to fill this gap by providing a rigorous experimental evaluation of the state-of-the-art Machine Learning (ML) and Deep Learning (DL) algorithms for TSC tasks in manufacturing and industrial settings. We first explored and compiled a comprehensive list of more than 92 state-of-the-art algorithms from both TSC and manufacturing literature. Following this, we methodologically selected the 36 most representative algorithms from this list. To evaluate their performance across various manufacturing classification tasks, we curated a set of 22 manufacturing datasets, representative of different characteristics that cover diverse manufacturing problems. Subsequently, we implemented and evaluated the algorithms on the manufacturing benchmark datasets, and analyzed the results for each dataset. Based on the results, ResNet, DrCIF, InceptionTime, and ARSENAL emerged as the top-performing algorithms, boasting an average accuracy of over 96.6 % across all 22 manufacturing TSC datasets. These findings underscore the robustness, efficiency, scalability, and effectiveness of convolutional kernels in capturing temporal features in time-series data collected from manufacturing systems for TSC tasks, as three out of the top four performing algorithms leverage these kernels for feature extraction. Additionally, LSTM, BiLSTM, and TS-LSTM algorithms deserve recognition for their effectiveness in capturing features within manufacturing time-series data using RNN-based structures.