A multi-sensor algorithm for activity and workflow recognition in an industrial setting

Abstract

In the recent revival of human labour in industry, and the subsequent push to optimally combine the strengths of man and machine in industrial processes, there is an increased need for methods allowing machines to understand and interpret the actions of their users. An important aspect of this is the understanding and evaluation of the progress of the workflows that are to be executed. Methods for this require both an appropriate choice of sensors, as well as algorithms capable of quickly and efficiently evaluating activity and workflow progress. In this paper we present such an algorithm, which provides activity and workflow recognition using both depth and RGB cameras as input. The algorithm's main purpose is to be used in an industrial training station, allowing novice workers to learn the necessary steps in assembling nordic ski products without the need for human supervision. We will describe how the algorithm recognizes predefined workflows in the sensor data, and present a comprehensive evaluation of the algorithm's performance on a real data recording of operators performing their work in an industrial setting. We will show that the algorithm fulfills the necessary requirements and is ready to be implemented in the training station application.