Agricultural Machinery Operator Monitoring System (Ag-OMS): A Machine Learning Approach for Real-Time Operator Safety Assessment

Highlights A machine learning-based real-time monitoring system for agricultural machinery operators was developed. Categorization of tractor operators’ behaviors in real-time into low, medium, and high-risk safety behaviors. Visual and sound feedback alert system of Ag-OMS triggered when operators engaged in unsafe operating behaviors. ABSTRACT. The 2015 CS-CASH (Central States Center for Agricultural Safety and Health, 2015) Injury Surveillance Surveys showed that around 19% of injuries to agricultural producers are related to tractors or large agricultural machinery, yet only a limited number of studies are found that address tools and methods for monitoring safety behaviors of agricultural machinery operators in real-time. The current safety behavior monitoring approaches require an in-person presence, which can be both time- and cost-inefficient, and the other available methods lack a feedback element to alert operators in real-time. As a result, the research presented in this study aimed to develop an automated approach to monitoring tractor operators' safety behaviors through the use of a trained machine learning (ML) model and a feedback system to alert operators when they engage in unsafe practices. For the ML model development, a skeleton-detecting algorithm called OpenPose was used to detect real-time human postures in a livestreaming video feed from a camera installed in the tractor cab. The model was then trained on three separate categories of tractor operators’ safety operating behaviors, and this trained classifier was used to label operators’ safety behaviors in real time based on the three safety classes. A feedback mechanism controlled by an onboard microcontroller was then used to alert the operators when unsafe operating behavior was detected to facilitate safe practices. This monitoring system, named Ag-OMS (Agricultural Machinery Operators Monitoring System), monitored the ingress/egress operators’ behaviors in real-time entering and exiting the tractor cab. The Ag-OMS successfully identified the ingress/egress operators’ behaviors with an accuracy of 97% on the testing datasets for all safety risk categories. Keywords: Ag-OMS, Machine learning (ML), Safety behaviors, OpenPose, Tractor operator.