Survey and Performance Test of Python-based Libraries for Parallel Processing

Abstract

By the Fourth Industrial Revolution and the 10 strategic technology of the Gartner Group, Artificial Intelligence(AI) technology was important and affected many areas. One of the ways to accelerate AI services is the Python-based parallel processing library. High-level programming languages such as Python are increasingly used to provide intuitive interfaces to libraries written in lower-level languages and for assembling applications from various components. This migration towards orchestration rather than implementation, coupled with the growing need for parallel computing (e.g., due to big data and the end of Moore's law), necessitates rethinking how parallelism is expressed in programs.[1] In this paper, take a look at a Python-based distributed parallel processing library, one of the ways to accelerate AI services, and use it to compare serial and parallel processing times.