Community Research Earth Digital Intelligence Twin: a scalable framework for AI-driven Earth System Modeling

Abstract

Recent advancements in artificial intelligence (AI) numerical weather prediction (NWP) have transformed atmospheric modeling. AI NWP models outperform state-of-the-art conventional NWP models like the European Center for Medium Range Weather Forecasting’s (ECMWF) Integrated Forecasting System (IFS) on several global metrics while requiring orders of magnitude fewer computational resources. However, existing AI NWP models still face limitations due to training datasets and dynamic timestep choices, often leading to artifacts that affect performance. To begin to address these challenges, we introduce the Community Research Earth Digital Intelligence Twin (CREDIT) framework, developed at the NSF National Center for Atmospheric Research (NCAR). CREDIT is a flexible, scalable, foundational research platform for training and deploying AI NWP models, providing an end-to-end pipeline for data preprocessing, model training, and evaluation. The CREDIT framework supports both existing architectures and the development of new models. We showcase this flexibility with WXFormer, a novel multiscale vision transformer designed to predict atmospheric states while mitigating common AI NWP pitfalls through techniques like spectral normalization, intelligent padding, and multi-step training. Additionally, we train the FuXi architecture within the CREDIT framework for comparison. Our results demonstrate that both FuXi and WXFormer, trained on hybrid sigma-pressure level ERA5 sampled at 6-h intervals, generally achieve better performance than the IFS High-Resolution (IFS HRES) on 10-day forecasts, offering potential improvements in efficiency and accuracy. The modular nature of CREDIT fosters collaboration, enabling researchers to experiment with models, datasets, and training options.