A Heuristic ETL Process to Dynamically Separate and Compress AIS Data

Abstract

Massive vessel trajectory data can be obtained from marine Automatic Identification Systems (AIS) to extract information about water traffic. To efficiently collect and process such a huge amount of data special methods are needed. This study designs a new system for collecting and processing AIS data in a real-time manner. The proposed system not only compresses vessel data while keeping useful information but also adds more attributes to raw trajectory data. The additional attributes include trip id, trip origin/destination, traffic density, and traffic flow. At first, this study presents a dynamic Extract, Transform, and Load (ETL) pipeline that collects AIS messages from vessels, processes those raw data, and loads the processed data in a central database. An optimized algorithm is developed that can process millions of records as fast as possible and send the processed data to production. Next, a user interface is developed to quantify traffic conditions and visualize them in graphs and maps. Finally, Gulf Intercoastal Waterway (GIWW) is considered as study area, where historical and real-time AIS data located in GIWW were collected to test the functionality of the method.