Webcam-based categorization of task engagement of PC users at work

Abstract

In this paper, we propose a support method for PC users to monitor their task engagement. Our approach is based on the number of keyboard strokes and pixel changes on the screen, and images from an ordinary webcam equipped with a PC. With our system, supervisors or teachers would improve their quality of guidance and instructions given at the right moment when workers or students require some support due to reasons such as slow progress and technical difficulties. In conventional methods, a special device such as an acceleration sensor for each individual is often required to acquire information on one’s working status and body movements, which is difficult to deploy in a real environment due to its cost for sensors. A face detection method based on Deep Neural Network, such as SSD, allows as to implement a cheaper system using an ordinary web camera. We calculate the average difference between two grabbed frames from the user’s screen to estimate the amount of screen changes between a given time interval. The number of key strokes typed by the user is another factor to estimate their task engagement. These factors are used to categorize the work mode of users. We use the K-Means method based on the Euclidean distance to cluster the recorded factors to determine thresholds for task categorization. We conducted experiments seven participants to evaluate the accuracy of our categorization method. Every participant is asked to categorize 15 scenes into four work modes. A scene includes a camera image with the PC user’s face, the screenshot path the moment, and the number of key strokes. The results from these participants are then compared with those of our system that categorized the same scenes with the thresholds on three factors. Approximately only 60% of these results matched each other, where we have enough room to improve our approach. Future work includes the selection of features that are far more effective for categorization, a better estimation of pixel changes on the PC screen, and evaluation experiments with more participants.