A semantic real-time activity recognition system for sequential procedures in vocational learning

In various areas of study, standard established procedures are critical for the successful accomplishment of a kinaesthetic task. Such standard procedures are important in various industries like engineering and health. This study makes a case for the development of intelligent activity monitoring systems for learning purposes through a proof of concept in first-aid training. Minor accidents such as simple cuts, bruises and minor burns are frequently treated without the need of emergency medical services. However, an incorrect first-aid procedure may lead to medical complications. This study aims to aid a learner to train how to perform a first-aid procedure for treating a wound through real-time monitoring, instructions and feedback. We propose a three-phase system where fast object detection, activity recognition in a temporal dimension and sequencing are used to semantically understand leaner actions. The You Only Look Once (YOLOv5) was used in phase 1 to detect multiple objects like wounds and bandages and Mediapipe to detect hand landmarks. Each class was assigned a different threshold for more accurate detections. The object detection model achieved a mean Average Precision (mAP) of 72.74% on the validation set and was subsequently used in a temporal manner to recognize an action. This temporal method to recognize the action of applying pressure over a wound, achieved an F1-Score of 91.67%. The method using an ontology-based technique to recognize the action of applying a bandage, achieved an F1-Score of 90.91%. The optimum distance from camera was found to be the actor placed at a position where the arm of the wounded actor occupies a significant portion of the viewport, whilst the optimum camera angle was found to be 110°. The created sequencing algorithm was tested using three different scenarios with the aid of a number of participants. The overall accuracy was 83.33%, wherein the result highlights that the algorithm is able to identify the sequence being conducted even with minimal movement involved during bandage application. The proposed system has high prospects of addressing challenges in a real-world environment.