Building an Inference Server Platform for Large Language Models Using Dataflow PIM Platform

Abstract

Processing-in-Memory (PIM) has garnered attention as a platform for large language model inference due to its ability to perform computations within memory, leveraging the internal bandwidth of memory components. In data center environments, to execute AI models across multiple nodes, an inference server is typically deployed at the data center's frontend. This server orchestrates the assignment of AI inference tasks to the appropriate nodes. This paper presents the construction of an open source-based inference server designed for easy deployment of a PIM platform grounded in data flow architecture within a data center setting. We have conducted operational tests on large language models to validate the efficacy of our approach.