Software development framework for a distributed storage and GPGPU data processing infrastructure

Abstract

The problem of choosing the cluster or a cluster node for task execution is important for the overall performance of a distributed system. This paper presents a complex approach to the planning of computations on heterogeneous distributed systems - a set of clusters and NoSQL storage systems. Dynamic scheduling algorithm depends on: the inter-cluster network parameters, characteristics of cluster interconnect, compute nodes utilization, co-processors computing capabilities, etc. In this work Hadoop YARN, CUDA technology and NoSQL-system Apache Cassandra has been used as the experimental platform.