A peer-to-peer infrastructure for distributing large scientific data sets over wide area high-performance networks: experimental studies using wide area layer 2 services

Abstract

This paper presents Sector, a distributed environment that was created specifically to address the challenges inherent in accessing, exploring, analyzing and transporting extremely large scientific datasets over high performance wide area networks. To date, the effective utilization of such datasets has been limited because accessing and transporting large remote data sets in traditional distributed computing environments is often a challenge. Sector was designed and developed to eliminate these barriers. The Sector architecture incorporates specialized communications services and specialized data services that are designed for high volume data flows over wide area high performance optical networks. This design employs several innovative techniques to ensure that data flows are maximized at all times and at all locations required regardless of distances. This architectural design has been implemented on a prototype international experimental testbed. The results and implications of these preliminary tests are described.