RouteReplies: Alleviating Long Latency in Many-Chip-Module GPUs

GPU chip module count is expected to keep increasing to meet the strong scaling demands of parallel applications. In many-chip-module GPUs, memory access latency seriously limits the performance since the transferring latency between different GPU modules is very high, which cannot be easily hidden by switching between different ready threads. To handle this problem, we propose RouteReplies, which enables a GPU module to fetch data from other GPU modules in the routing path. Leveraging the data locality between different GPU modules, RouteReplies significantly reduces the memory access latency since the memory request does not need to fetch data from the faraway memory partition. For a set of applications exhibiting varying degrees of inter-module locality, RouteReplies reduces memory access latency and increases performance by 54.8% on average (up to 364.8%).