An open compute and data federation as an alternative to monolithic infrastructures for big Earth data analytics

Abstract

ABSTRACT An adequate compute and storage infrastructure supporting the full exploitation of Copernicus and Earth Observation datasets is currently not available in Europe. This paper presents the cross-disciplinary open-source technologies being leveraged in the C-SCALE project to develop an open federation of compute and data providers as an alternative to monolithic infrastructures for processing and analysing Copernicus and Earth Observation data. Three critical aspects of the federation and the chosen technologies are elaborated upon: (1) federated data discovery, (2) federated access and (3) software distribution. With these technologies the open federation aims to provide homogenous access to resources, thereby enabling its users to generate meaningful results quickly and easily. This will be achieved by abstracting the complexity of infrastructure resource access provisioning and orchestration, including discovery of data across distributed archives, away from the end-users. Which is needed because end-users wish to focus on analysing ready-to-use data products and models rather than spending their time on the setup and maintenance of complex and heterogeneous IT infrastructures. The open federation will support processing and analysing the vast amounts of Copernicus and Earth Observation data that are critical for the implementation of the Destination Earth resp. Digital Twins vision for a high precision digital model of the Earth to model, monitor and simulate natural phenomena and related human activities.