Understanding the Performance of a Prototype of a WLCG Data Lake for HL-LHC

Abstract

Storage is identified as one of the main challenges for WLCG in the next decade in the computing strategy document for HL-LHC [1]. Extrapolating todays computing models, the ATLAS and CMS experiments alone would need one order of magnitude more storage resources than what could be provided by the funding agencies. Organization and consolidation of storage and evolution of the compute facilities will be central in addressing the possible resources shortage. In this contribution we describe the architecture of a prototype of a WLCG data lake for HL-LHC. A WLCG data lake aims to provide a geographically distributed storage service, distributed across large data centres interconnected by fast networks with low latency. We present methodology used to measure and understand the performance of a WLCG data lake prototype, in order to compare event throughput at the same cost at the same compute facilities backed by the traditional storage services and backed by the WLCG data lake. We will discuss various possible data processing models w.r.t. network latency, available storage media, and data caching approaches. We will present benchmarks for storage service and compute performance.