A comparison of machine learning classifiers in predicting safety for a multi-component dynamic system representation of an autonomous vessel

Abstract

To operate autonomously for weeks to months at a time, crewless vessels must be able to integrate uncertain estimates of sub-system health and weather forecasts into operational plans. To date, most digital twins have focused on estimating the health of single sub-systems, not the multi-system integration problem. This work establishes a four-component integration problem using real-world weather forecasts from the European Centre for Medium-Range Weather Forecast and a simulated linear spring–mass–damper system excited by wave elevation. Each component in the spring–mass–damper system is monitored with techniques of differing accuracies representative of marine-type health uncertainties. Weather forecast uncertainty is included using weather predictions of significant wave height and peak period up to 10 days out. As well as their exact values, different test cases include the spring, mass, and damper being modeled as noisy sensors representative of sensors onboard a vessel, as well as the spring being modeled as a visually-inspected system component, reflective of human impact onboard a vessel. The performance of five established machine learning classification algorithms, Support Vector Machines, K Nearest Neighbors, Decision Trees, Random Forest, and XGBoost in assessing if the spring–mass–damper system will exceed a displacement threshold is then evaluated for 1000 physical locations under the differing uncertainty assumptions. Overall, even for the simple linear system considered here, none of the existing methods perform with high accuracy when subject to common types of uncertainty, with the SVM system generally performing best and the DT system performing worse. Conclusions and discussions for improving system-level synthesis are made.