Research on ship data analysis based on Spark platform

Water transportation is a transportation mode with ships as the main means. China is one of the most important shipping countries in the world and the largest shipbuilding country. Ship basic data is the core of management by maritime regulatory authorities at all levels. With the development of the new generation of information technology, the shipping industry has entered the 4.0 era. It puts forward higher requirements for maritime supervision. In order to improve the level of maritime supervision and enable maritime management departments to timely grasp ship information, a ship data analysis platform based on Spark platform is designed and implemented. The platform extracts data from the business database into the Hive data warehouse through Sqoop, analyzes ship data through the Spark distributed computing platform, stores the analysis results in the MySQL database, extracts ship basic data from the database through the Spring Boot framework, obtains JSON data transmitted in the background using Axios, binds data through Vue.js, and displays charts through Echarts and Mapbox.