N-DISE: NDN-based data distribution for large-scale data-intensive science

Abstract

To meet unprecedented challenges faced by the world's largest data- and network-intensive science programs, we design and implement a new, highly efficient and field-tested data distribution, caching, access and analysis system for the Large Hadron Collider (LHC) high energy physics (HEP) network and other major science programs. We develop a hierarchical Named Data Networking (NDN) naming scheme for HEP data, implement new consumer and producer applications to interface with the high-performance NDN-DPDK forwarder, and build on recently developed high-throughput NDN caching and forwarding methods. We integrate NDN systems concepts and algorithms with the mainstream data distribution, processing, and management system of the Compact Muon Solenoid (CMS) experiment. We design and prototype stable, high-performance virtual LANs (VLANs) over a continental-scale wide area network testbed. In extensive experiments, our proposed integrated system, named NDN for Data-Intensive Science Experiments (N-DISE), is shown to deliver LHC data over the wide area network (WAN) testbed at throughputs exceeding 31 Gbps between Caltech and StarLight, with dramatically reduced download time.