Connecting arbitrary data resources to the grid

Abstract

Many scientific grid systems have been running and serving researchers for many years around the world. Among them, Globus Toolkit and its variants are playing an important role as the basis of most of those existing grid systems. However, the way data is stored and accessed varies. Proprietary protocols have been designed and developed to serve data by different storage systems or file systems. One example is the integrated Rule Oriented Data System (iRODS), which is a data grid system with the non-standard iRODS protocol and has its own client tools and API. Consequently, it is difficult for the grid to connect to it directly and stage data to computers in the grid for processing. It is usually an ad hoc process to transfer data between two data systems with different protocols. In addition, existing data transfer services are mostly designed for the grid and do not understand proprietary protocols. This requires users to transfer data from the source to a temporary space, and then transfer it from the temporary space to the destination, which is complex, inefficient and error-prone. Some work has been done on the client side to address this issue. In order to address the issues of data staging and data transfer in one solution, this paper describes a different but easy and generic approach to connect any data systems to the grid, by providing a service with an abstract framework to convert any underlying data system protocol to the GridFTP protocol, a de facto standard of data transfer for the grid.