Ensemble of Vision Transformers for Sea State Classification

Abstract

This study presents an innovative method for real-time sea state classification that utilizes an ensemble-based computer vision framework. Multiple pretrained submodels, each individually fine-tuned on specialized data sets, are integrated to recognize eight classes on the Beaufort scale. A comprehensive multimodal data set was assembled, combining visual information with wind speed data from anemometers and further optimized through preprocessing steps. Standalone models based on cutting-edge architectures, such as the data efficient vision transformer and EfficientNet deep convolutional network, were adapted specifically for sea state recognition. Initial results yielded F1 scores of 66% and 65%, respectively, for these models. To boost overall accuracy, a weighted majority voting and precision-driven ensemble strategy was adopted. The unimodal ensemble achieved 73% F1 score, while the multimodal ensemble, leveraging both image and wind data, reached an F1 score of 97%. For a reduced six-class scenario, where noncritical classes were grouped, the unimodal ensemble reached an impressive F1 score of 94%. These outcomes illustrate the potential of this ensemble approach to significantly advance maritime safety and operational efficiency in sea state monitoring.