Proceedings of the 37th International Conference on Supercomputing最新文献_第3页

Multi-GPU Communication Schemes for Iterative Solvers: When CPUs are Not in Charge 迭代求解器的多gpu通信方案:当cpu不负责时

Proceedings of the 37th International Conference on Supercomputing Pub Date : 2023-06-21 DOI: 10.1145/3577193.3593713

Ismayil Ismayilov, Javid Baydamirli, Doğan Sağbili, M. Wahib, D. Unat

{"title":"Multi-GPU Communication Schemes for Iterative Solvers: When CPUs are Not in Charge","authors":"Ismayil Ismayilov, Javid Baydamirli, Doğan Sağbili, M. Wahib, D. Unat","doi":"10.1145/3577193.3593713","DOIUrl":"https://doi.org/10.1145/3577193.3593713","url":null,"abstract":"This paper proposes a fully autonomous execution model for multi-GPU applications that completely excludes the involvement of the CPU beyond the initial kernel launch. In a typical multi-GPU application, the host serves as the orchestrator of execution by directly launching kernels, issuing communication calls, and acting as a synchronizer for devices. We argue that this orchestration, or control flow path, causes undue overhead and can be delegated entirely to devices to improve performance in applications that require communication among peers. For the proposed CPU-free execution model, we leverage existing techniques such as persistent kernels, thread block specialization, device-side barriers, and device-initiated communication routines to write fully autonomous multi-GPU code and achieve significantly reduced communication overheads. We demonstrate our proposed model on two broadly used iterative solvers, 2D/3D Jacobi stencil and Conjugate Gradient(CG). Compared to the CPU-controlled baselines, the CPU-free model can improve 3D stencil communication latency by 58.8% and provide a 1.63x speedup for CG on 8 NVIDIA A100 GPUs. The project code is available at https://github.com/ParCoreLab/CPU-Free-model.","PeriodicalId":424155,"journal":{"name":"Proceedings of the 37th International Conference on Supercomputing","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122836302","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Parallel Software for Million-scale Exact Kernel Regression 百万尺度精确核回归并行软件

Proceedings of the 37th International Conference on Supercomputing Pub Date : 2023-06-21 DOI: 10.1145/3577193.3593737

Yu Chen, Lucca Skon, James R. McCombs, Zhenming Liu, A. Stathopoulos

{"title":"Parallel Software for Million-scale Exact Kernel Regression","authors":"Yu Chen, Lucca Skon, James R. McCombs, Zhenming Liu, A. Stathopoulos","doi":"10.1145/3577193.3593737","DOIUrl":"https://doi.org/10.1145/3577193.3593737","url":null,"abstract":"We present the design and the implementation of a kernel principal component regression software that handles training datasets with a million or more observations. Kernel regressions are nonlinear and interpretable models that have wide downstream applications, and are shown to have a close connection to deep learning. Nevertheless, the exact regression of large-scale kernel models using currently available software has been notoriously difficult because it is both compute and memory intensive and it requires extensive tuning of hyperparameters. While in computational science distributed computing and iterative methods have been a mainstay of large scale software, they have not been widely adopted in kernel learning. Our software leverages existing high performance computing (HPC) techniques and develops new ones that address cross-cutting constraints between HPC and learning algorithms. It integrates three major components: (a) a state-of-the-art parallel eigenvalue iterative solver, (b) a block matrix-vector multiplication routine that employs both multi-threading and distributed memory parallelism and can be performed on-the-fly under limited memory, and (c) a software pipeline consisting of Python front-ends that control the HPC backbone and the hyperparameter optimization through a boosting optimizer. We perform feasibility studies by running the entire ImageNet dataset and a large asset pricing dataset.","PeriodicalId":424155,"journal":{"name":"Proceedings of the 37th International Conference on Supercomputing","volume":"4 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124346485","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Performance Embeddings: A Similarity-Based Transfer Tuning Approach to Performance Optimization 性能嵌入:一种基于相似性的性能优化转移调优方法

Proceedings of the 37th International Conference on Supercomputing Pub Date : 2023-06-21 DOI: 10.1145/3577193.3593714

Lukas Trümper, Tal Ben-Nun, Philipp Schaad, A. Calotoiu, T. Hoefler

{"title":"Performance Embeddings: A Similarity-Based Transfer Tuning Approach to Performance Optimization","authors":"Lukas Trümper, Tal Ben-Nun, Philipp Schaad, A. Calotoiu, T. Hoefler","doi":"10.1145/3577193.3593714","DOIUrl":"https://doi.org/10.1145/3577193.3593714","url":null,"abstract":"Performance optimization is an increasingly challenging but often repetitive task. While each platform has its quirks, the underlying code transformations rely on data movement and computational characteristics that recur across applications. This paper proposes to leverage those similarities by constructing an embedding space for subprograms. The continuous space captures both static and dynamic properties of loop nests via symbolic code analysis and performance profiling, respectively. Performance embeddings enable direct knowledge transfer of performance tuning between applications, which can result from autotuning or tailored improvements. We demonstrate this transfer tuning approach on case studies in deep neural networks, dense and sparse linear algebra compositions, and numerical weather prediction stencils. Transfer tuning reduces the search complexity by up to four orders of magnitude and outperforms the MKL library in sparse-dense matrix multiplication. The results exhibit clear correspondences between program characteristics and optimizations, outperforming prior specialized state-of-the-art approaches and generalizing beyond their capabilities.","PeriodicalId":424155,"journal":{"name":"Proceedings of the 37th International Conference on Supercomputing","volume":"33 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133933653","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

GRAP: Group-level Resource Allocation Policy for Reconfigurable Dragonfly Network in HPC 高性能计算中可重构蜻蜓网络的组级资源分配策略

Proceedings of the 37th International Conference on Supercomputing Pub Date : 2023-06-21 DOI: 10.1145/3577193.3593732

Guangnan Feng, Dezun Dong, Shizhen Zhao, Yutong Lu

{"title":"GRAP: Group-level Resource Allocation Policy for Reconfigurable Dragonfly Network in HPC","authors":"Guangnan Feng, Dezun Dong, Shizhen Zhao, Yutong Lu","doi":"10.1145/3577193.3593732","DOIUrl":"https://doi.org/10.1145/3577193.3593732","url":null,"abstract":"Dragonfly is a highly scalable, low-diameter, and cost-efficient network topology, which has been adopted in new exascale High Performance Computing (HPC) systems. However, Dragonfly topology suffers from the limited direct links between groups. The reconfigurable network can solve this problem by reconfiguring topology to adjust the number of direct links between groups. While the performance improvement of a single job on reconfigurable HPC network has been evaluated in previous works, the performance of HPC workloads has not been studied because of the lack of an appropriate resource allocation policy. In this work, we propose Group-level Resource Allocation Policy (GRAP) to allocate both compute nodes and Reconfigurable Links for jobs in Reconfigurable Dragonfly Network (RDN). We start with formulating three design principles: reconfigurable network should be reconfiguration interference-free, guarantee connectivity and performance for each job, and satisfy varied resource requests. According to the principles, GRAP uses different strategies for small and large jobs, and contains three allocation modes for large jobs: Balance Mode, Custom Mode, and Adaptive Mode. Finally, we evaluate GRAP with the CODES network simulation framework and the Slurm Simulator using real workload traces. The results demonstrate that RDN coupled with GRAP achieves lower latency, higher bandwidth, and lower job wait time.","PeriodicalId":424155,"journal":{"name":"Proceedings of the 37th International Conference on Supercomputing","volume":"22 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114838903","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

FLORIA: A Fast and Featherlight Approach for Predicting Cache Performance FLORIA:预测缓存性能的快速轻量级方法

Proceedings of the 37th International Conference on Supercomputing Pub Date : 2023-06-21 DOI: 10.1145/3577193.3593740

Jun Xiao, Yaocheng Xiang, Xiaolin Wang, Yingwei Luo, A. Pimentel, Zhenlin Wang

{"title":"FLORIA: A Fast and Featherlight Approach for Predicting Cache Performance","authors":"Jun Xiao, Yaocheng Xiang, Xiaolin Wang, Yingwei Luo, A. Pimentel, Zhenlin Wang","doi":"10.1145/3577193.3593740","DOIUrl":"https://doi.org/10.1145/3577193.3593740","url":null,"abstract":"The cache Miss Ratio Curve (MRC) serves a variety of purposes such as cache partitioning, application profiling and code tuning. In this work, we propose a new metric, called cache miss distribution, that describes cache miss behavior over cache sets, for predicting cache MRCs. Based on this metric, we present FLORIA, a software-based, online approach that approximates cache MRCs on commodity systems. By polluting a tunable number of cache lines in some selected cache sets using our designed microbenchmark, the cache miss distribution for the target workload is obtained via hardware performance counters with the support of precise event based sampling (PEBS). A model is developed to predict the MRC of the target workload based on its cache miss distribution. We evaluate FLORIA for systems consisting of a single application as well as a wide range of different workload mixes. Compared with the state-of-the-art approaches in predicting online MRCs, FLORIA achieves the highest average accuracy of 97.29% with negligible overhead. It also allows fast and accurate estimation of online MRC within 5ms, 20X faster than the state-of-the-art approaches. We also demonstrate that FLORIA can be applied to guiding cache partitioning for multiprogrammed workloads, helping to improve overall system performance.","PeriodicalId":424155,"journal":{"name":"Proceedings of the 37th International Conference on Supercomputing","volume":"263 1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114233757","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

Roar: A Router Microarchitecture for In-network Allreduce 面向网络内Allreduce的路由器微架构

Proceedings of the 37th International Conference on Supercomputing Pub Date : 2023-06-21 DOI: 10.1145/3577193.3593711

Ruiqi Wang, Dezun Dong, Fei Lei, Junchao Ma, Ketong Wu, KaiCheng Lu

引用次数: 0

BiRFIA: Selective Binary Rewriting for Function Interception on ARM 基于ARM的函数拦截的选择性二进制重写

Proceedings of the 37th International Conference on Supercomputing Pub Date : 2023-06-21 DOI: 10.1145/3577193.3593701

Kelun Lei, Xin You, Hailong Yang, Zhongzhi Luan, D. Qian

{"title":"BiRFIA: Selective Binary Rewriting for Function Interception on ARM","authors":"Kelun Lei, Xin You, Hailong Yang, Zhongzhi Luan, D. Qian","doi":"10.1145/3577193.3593701","DOIUrl":"https://doi.org/10.1145/3577193.3593701","url":null,"abstract":"Function interception of fully-optimized binaries is widely used for optimization with its ability to accurately collect runtime information and detect inefficiencies at the function level. However, the implementation of function interception with existing binary rewriting techniques still suffers from limited reliability and performance on ARM platform. In this paper, we propose BiRFIA, an efficient selective binary rewriting framework for function interception targeting highly optimized binaries on ARM platforms. BiRFIA performs static binary rewriting of specific functions and intercepts them through well-formed trampoline sections and external instrumentation libraries. Besides, BiRFIA places complex instrumentation code in the trampoline section and jumps to the trampoline section via an adaptive instruction eviction strategy, which significantly reduces the probability of unexpected errors. For evaluation, we develop two function interception tools based on BiRFIA, including a function performance event counter collector and a function parameter tracer. Guided by these tools, we optimize several benchmarks and real-world programs, yielding up to 8% performance speedup. Our evaluation result demonstrates that BiRFIA incurs negligible runtime overhead of 1.006× on average.","PeriodicalId":424155,"journal":{"name":"Proceedings of the 37th International Conference on Supercomputing","volume":"125 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-06-21","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"130115530","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

RT-kNNS Unbound: Using RT Cores to Accelerate Unrestricted Neighbor Search RT- knns Unbound:使用RT内核加速无限制邻居搜索

Proceedings of the 37th International Conference on Supercomputing Pub Date : 2023-05-26 DOI: 10.1145/3577193.3593738

Vani Nagarajan, D. Mandarapu, Milind Kulkarni

{"title":"RT-kNNS Unbound: Using RT Cores to Accelerate Unrestricted Neighbor Search","authors":"Vani Nagarajan, D. Mandarapu, Milind Kulkarni","doi":"10.1145/3577193.3593738","DOIUrl":"https://doi.org/10.1145/3577193.3593738","url":null,"abstract":"The problem of identifying the k-Nearest Neighbors (kNNS) of a point has proven to be very useful both as a standalone application and as a subroutine in larger applications. Given its far-reaching applicability in areas such as machine learning and point clouds, extensive research has gone into leveraging GPU acceleration to solve this problem. Recent work has shown that using Ray Tracing cores in recent GPUs to accelerate kNNS is much more efficient compared to traditional acceleration using shader cores. However, the existing translation of kNNS to a ray tracing problem imposes a constraint on the search space for neighbors. Due to this, we can only use RT cores to accelerate fixed-radius kNNS, which requires the user to set a search radius a priori and hence can miss neighbors. In this work, we propose TrueKNN, the first unbounded RT-accelerated neighbor search. TrueKNN adopts an iterative approach where we incrementally grow the search space until all points have found their k neighbors. We show that our approach is orders of magnitude faster than existing approaches and can even be used to accelerate fixed-radius neighbor searches.","PeriodicalId":424155,"journal":{"name":"Proceedings of the 37th International Conference on Supercomputing","volume":"19 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-26","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114675088","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0

FMI: Fast and Cheap Message Passing for Serverless Functions FMI:无服务器功能的快速和廉价的消息传递

Proceedings of the 37th International Conference on Supercomputing Pub Date : 2023-05-15 DOI: 10.1145/3577193.3593718

Marcin Copik, Roman Böhringer, A. Calotoiu, T. Hoefler

{"title":"FMI: Fast and Cheap Message Passing for Serverless Functions","authors":"Marcin Copik, Roman Böhringer, A. Calotoiu, T. Hoefler","doi":"10.1145/3577193.3593718","DOIUrl":"https://doi.org/10.1145/3577193.3593718","url":null,"abstract":"Serverless functions provide elastic scaling and a fine-grained billing model, making Function-as-a-Service (FaaS) an attractive programming model. However, for distributed jobs that benefit from large-scale and dynamic parallelism, the lack of fast and cheap communication is a major limitation. Individual functions cannot communicate directly, group operations do not exist, and users resort to manual implementations of storage-based communication. This results in communication times multiple orders of magnitude slower than those found in HPC systems. We overcome this limitation and present the FaaS Message Interface (FMI). FMI is an easy-to-use, high-performance framework for general-purpose point-to-point and collective communication in FaaS applications. We support different communication channels and offer a model-driven channel selection according to performance and cost expectations. We model the interface after MPI and show that message passing can be integrated into serverless applications with minor changes, providing portable communication closer to that offered by high-performance systems. In our experiments, FMI can speed up communication for a distributed machine learning FaaS application by up to 162x, while simultaneously reducing cost by up to 397 times.","PeriodicalId":424155,"journal":{"name":"Proceedings of the 37th International Conference on Supercomputing","volume":"3 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127449566","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 4

Revisiting Temporal Blocking Stencil Optimizations 重新审视时态阻塞模板优化

Proceedings of the 37th International Conference on Supercomputing Pub Date : 2023-05-12 DOI: 10.1145/3577193.3593716

Lingqi Zhang, M. Wahib, Peng Chen, Jintao Meng, Xiao Wang, Toshio Endo, S. Matsuoka

{"title":"Revisiting Temporal Blocking Stencil Optimizations","authors":"Lingqi Zhang, M. Wahib, Peng Chen, Jintao Meng, Xiao Wang, Toshio Endo, S. Matsuoka","doi":"10.1145/3577193.3593716","DOIUrl":"https://doi.org/10.1145/3577193.3593716","url":null,"abstract":"Iterative stencils are used widely across the spectrum of High Performance Computing (HPC) applications. Many efforts have been put into optimizing stencil GPU kernels, given the prevalence of GPU-accelerated supercomputers. To improve the data locality, temporal blocking is an optimization that combines a batch of time steps to process them together. Under the observation that GPUs are evolving to resemble CPUs in some aspects, we revisit temporal blocking optimizations for GPUs. We explore how temporal blocking schemes can be adapted to the new features in the recent Nvidia GPUs, including large scratchpad memory, hardware prefetching, and device-wide synchronization. We propose a novel temporal blocking method, EBISU, which champions low device occupancy to drive aggressive deep temporal blocking on large tiles that are executed tile-by-tile. We compare EBISU with state-of-the-art temporal blocking libraries: STENCILGEN and AN5D. We also compare with state-of-the-art stencil auto-tuning tools that are equipped with temporal blocking optimizations: ARTEMIS and DRSTENCIL. Over a wide range of stencil benchmarks, EBISU achieves speedups up to 2.53x and a geometric mean speedup of 1.49x over the best state-of-the-art performance in each stencil benchmark.","PeriodicalId":424155,"journal":{"name":"Proceedings of the 37th International Conference on Supercomputing","volume":"64 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2023-05-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123771875","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 0