A distributed backpropagation algorithm of neural networks on distributed-memory multiprocessors

A distributed backpropagation algorithm for a fully connected multilayered neural network on a distributed-memory multiprocessor system is presented. The neurons on each layer are partitioned into p disjoint sets, and each set is mapped on a processor of a p-processor system. The algorithm, the communication pattern among the processors, and their time/space complexities are investigated, and the theoretical upper bound on speedup is obtained. The experimental speedup obtained with the algorithm on a ring of 32 transputers, which confirms the model and analysis, is reported. It is found that the choice of processor interconnection topology does not influence the speedup ratio.<>