Towards full AI model lifecycle management on EuroHPC systems, experiences with AIFS for DestinE

On October 13 2023 ECMWF released the first alpha version of its artificial intelligence forecasting system, AIFS, ECMWFs data-driven forecasts model. This first release came just a few months after ECMWF started the development of this new model that highlights the increased efforts in the field of machine learning (ML) that ECMWF has been building over the last few years. This paper describes the use of AIFS on EuroHPC systems in the context of DestinE. The main focus is on performance benchmarks on the different EuroHPC systems available to DestinE but also very much on the deployment and use of the tools to support the model lifecycle management. EuroHPC systems have already proven to be of great value for DestinE and in this paper, we describe how we leverage these systems for artificial intelligence (AI) and ML models in DestinE. We are closely working with EuroHPC and EuroHPC hosting sites through co-design and the optimization of existing solutions to optimize the usage of these systems in every step of the lifecycle management for AI and ML models. The performance benchmarks of our models on several EuroHPC systems showed that the speedup is close to linear up to several thousand GPUs, but that for each EuroHPC system a different optimization strategy must be used to achieve that. For model lifecycle management we found that we can use our in-house developed, domain specific, framework on EuroHPC systems and highlight some specific modifications and future improvements for EuroHPC systems. W e a l s o provide implementation details and share our experiences on how to retrieve and collect provenance data and information from models running on EuroHPC systems using (external to the EuroHPC system deployed) cloud native frameworks. Although we describe solutions in this paper that are designed to support our specific requirements and context, we believe that proposed solutions, developments and implementation details can also bring value beyond the broader NWP community.