Proceedings of the Machine Learning on HPC Environments最新文献

筛选
英文 中文
An In-depth Performance Characterization of CPU- and GPU-based DNN Training on Modern Architectures 现代架构下基于CPU和gpu的深度神经网络训练的深入性能表征
Proceedings of the Machine Learning on HPC Environments Pub Date : 2017-11-12 DOI: 10.1145/3146347.3146356
A. Awan, H. Subramoni, D. Panda
{"title":"An In-depth Performance Characterization of CPU- and GPU-based DNN Training on Modern Architectures","authors":"A. Awan, H. Subramoni, D. Panda","doi":"10.1145/3146347.3146356","DOIUrl":"https://doi.org/10.1145/3146347.3146356","url":null,"abstract":"Traditionally, Deep Learning (DL) frameworks like Caffe, TensorFlow, and Cognitive Toolkit exploited GPUs to accelerate the training process. This has been primarily achieved by aggressive improvements in parallel hardware as well as through sophisticated software frameworks like cuDNN and cuBLAS. However, recent enhancements to CPU-based hardware and software has the potential to significantly enhance the performance of CPU-based DL training. In this paper, we provide a complete performance landscape of CPU- and GPU-based DNN training. We characterize performance of DNN training for AlexNet and ResNet-50 for a wide-range of CPU and GPU architectures including the latest Intel Xeon Phi (Knights Landing) processors and NVIDIA Pascal GPUs. We also present multi-node DNN training performance results for AlexNet and ResNet-50 using Intel Machine Learning Scaling (MLSL) Library and Intel-Caffe. In addition, we provide a CPU vs. GPU comparison for multi-node training using OSU-Caffe and Intel-Caffe. To the best of our knowledge, this is the first study that dives deeper into the performance of DNN training in a holistic manner yet provides an in-depth look at layer-wise performance for different DNNs. We provide multiple key insights: 1) Convolutions account for the majority of time (up to 83% time) consumed in DNN training, 2) GPU-based training continues to deliver excellent performance (up to 18% better than KNL) across generations of GPU hardware and software, and 3) Recent CPU-based optimizations like MKL-DNN and OpenMP-based thread parallelism leads to excellent speed-ups over under-optimized designs (up to 3.2X improvement for AlexNet training).","PeriodicalId":298024,"journal":{"name":"Proceedings of the Machine Learning on HPC Environments","volume":"23 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"127859432","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 45
Training distributed deep recurrent neural networks with mixed precision on GPU clusters 在GPU集群上训练混合精度的分布式深度递归神经网络
Proceedings of the Machine Learning on HPC Environments Pub Date : 2017-11-12 DOI: 10.1145/3146347.3146358
Alexey Svyatkovskiy, J. Kates-Harbeck, W. Tang
{"title":"Training distributed deep recurrent neural networks with mixed precision on GPU clusters","authors":"Alexey Svyatkovskiy, J. Kates-Harbeck, W. Tang","doi":"10.1145/3146347.3146358","DOIUrl":"https://doi.org/10.1145/3146347.3146358","url":null,"abstract":"In this paper, we evaluate training of deep recurrent neural networks with half-precision floats. We implement a distributed, data-parallel, synchronous training algorithm by integrating TensorFlow and CUDA-aware MPI to enable execution across multiple GPU nodes and making use of high-speed interconnects. We introduce a learning rate schedule facilitating neural network convergence at up to O(100) workers. Strong scaling tests performed on clusters of NVIDIA Pascal P100 GPUs show linear runtime and logarithmic communication time scaling for both single and mixed precision training modes. Performance is evaluated on a scientific dataset taken from the Joint European Torus (JET) tokamak, containing multi-modal time series of sensory measurements leading up to deleterious events called plasma disruptions, and the benchmark Large Movie Review Dataset [11]. Half-precision significantly reduces memory and network bandwidth, allowing training of state-of-the-art models with over 70 million trainable parameters while achieving a comparable test set performance as single precision.","PeriodicalId":298024,"journal":{"name":"Proceedings of the Machine Learning on HPC Environments","volume":"14 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125578381","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 55
Evolving Deep Networks Using HPC 利用高性能计算技术发展深度网络
Proceedings of the Machine Learning on HPC Environments Pub Date : 2017-11-12 DOI: 10.1145/3146347.3146355
Steven R. Young, Derek C. Rose, J. T. Johnston, W. T. Heller, T. Karnowski, T. Potok, R. Patton, G. Perdue, Jonathan Miller
{"title":"Evolving Deep Networks Using HPC","authors":"Steven R. Young, Derek C. Rose, J. T. Johnston, W. T. Heller, T. Karnowski, T. Potok, R. Patton, G. Perdue, Jonathan Miller","doi":"10.1145/3146347.3146355","DOIUrl":"https://doi.org/10.1145/3146347.3146355","url":null,"abstract":"While a large number of deep learning networks have been studied and published that produce outstanding results on natural image datasets, these datasets only make up a fraction of those to which deep learning can be applied. These datasets include text data, audio data, and arrays of sensors that have very different characteristics than natural images. As these \"best\" networks for natural images have been largely discovered through experimentation and cannot be proven optimal on some theoretical basis, there is no reason to believe that they are the optimal network for these drastically different datasets. Hyperparameter search is thus often a very important process when applying deep learning to a new problem. In this work we present an evolutionary approach to searching the possible space of network hyperparameters and construction that can scale to 18, 000 nodes. This approach is applied to datasets of varying types and characteristics where we demonstrate the ability to rapidly find best hyperparameters in order to enable practitioners to quickly iterate between idea and result.","PeriodicalId":298024,"journal":{"name":"Proceedings of the Machine Learning on HPC Environments","volume":"108 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"124674021","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 43
Optimizing Convolutional Neural Networks for Cloud Detection 优化卷积神经网络云检测
Proceedings of the Machine Learning on HPC Environments Pub Date : 2017-11-12 DOI: 10.1145/3146347.3146352
J. T. Johnston, Steven R. Young, David Hughes, R. Patton, D. White
{"title":"Optimizing Convolutional Neural Networks for Cloud Detection","authors":"J. T. Johnston, Steven R. Young, David Hughes, R. Patton, D. White","doi":"10.1145/3146347.3146352","DOIUrl":"https://doi.org/10.1145/3146347.3146352","url":null,"abstract":"Deep convolutional neural networks (CNNs) have become extremely popular and successful at a number of machine learning tasks. One of the great challenges of successfully deploying a CNN is designing the network: specifying the network topology (sequence of layer types) and configuring the network (setting all the internal layer hyper-parameters). There are a number of techniques which are commonly used to design the network. One of the most successful is a simple (but lengthy) random search. In this paper we demonstrate how a random search can be dramatically improved by a two-phase search. The first phase is a traditional random search on n network configurations. The second phase exploits a support vector machine to guide a second random search on N network configurations. We apply this technique to a dataset containing satellite imagery and demonstrate that we can, with very high accuracy, identify regions containing clouds which obscure the landscape below.","PeriodicalId":298024,"journal":{"name":"Proceedings of the Machine Learning on HPC Environments","volume":"42 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129885445","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 17
Towards Scalable Parallel Training of Deep Neural Networks 面向深度神经网络的可扩展并行训练
Proceedings of the Machine Learning on HPC Environments Pub Date : 2017-11-12 DOI: 10.1145/3146347.3146353
S. A. Jacobs, Nikoli Dryden, R. Pearce, B. V. Essen
{"title":"Towards Scalable Parallel Training of Deep Neural Networks","authors":"S. A. Jacobs, Nikoli Dryden, R. Pearce, B. V. Essen","doi":"10.1145/3146347.3146353","DOIUrl":"https://doi.org/10.1145/3146347.3146353","url":null,"abstract":"We propose a new framework for parallelizing deep neural network training that maximize the amount of data that is ingested by the training algorithm. Our proposed framework called Livermore Tournament Fast Batch Learning (LTFB) targets large-scale data problems. The LTFB approach creates a set of Deep Neural Network (DNN) models and trains each instance of these models independently and in parallel. Periodically each model selects another model to pair with, exchanges models, and then run a local tournament against held-out tournament datasets. The winning model is will continue training on the local training datasets. This new approach maximizes computation and minimizes amount of synchronization required in training deep neural network, a major bottleneck in existing synchronous deep learning algorithms. We evaluate our proposed algorithm on two HPC machines at Lawrence Livermore National Laboratory including an early access IBM Power8+ with NVIDIA Tesla P100 GPUs machine. Experimental evaluations of the LTFB framework on two popular image classification benchmark: CIFAR10 [18] and ImageNet [19], show significant speed up compared to the sequential baseline.","PeriodicalId":298024,"journal":{"name":"Proceedings of the Machine Learning on HPC Environments","volume":"56 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123505558","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 11
An Efficient Task-based All-Reduce for Machine Learning Applications 一个高效的基于任务的全约简机器学习应用
Proceedings of the Machine Learning on HPC Environments Pub Date : 2017-11-12 DOI: 10.1145/3146347.3146350
Zhenyu Li, James Davis, S. Jarvis
{"title":"An Efficient Task-based All-Reduce for Machine Learning Applications","authors":"Zhenyu Li, James Davis, S. Jarvis","doi":"10.1145/3146347.3146350","DOIUrl":"https://doi.org/10.1145/3146347.3146350","url":null,"abstract":"All-Reduce is a collective-combine operation frequently utilised in synchronous parameter updates in parallel machine learning algorithms. The performance of this operation - and subsequently of the algorithm itself - is heavily dependent on its implementation, configuration and on the supporting hardware on which it is run. Given the pivotal role of all-reduce, a failure in any of these regards will significantly impact the resulting scientific output. In this research we explore the performance of alternative all-reduce algorithms in data-flow graphs and compare these to the commonly used reduce-broadcast approach. We present an architecture and interface for all-reduce in task-based frameworks, and a parallelization scheme for object-serialization and computation. We present a concrete, novel application of a butterfly all-reduce algorithm on the Apache Spark framework on a high-performance compute cluster, and demonstrate the effectiveness of the new butterfly algorithm with a logarithmic speed-up with respect to the vector length compared with the original reduce-broadcast method - a 9x speed-up is observed for vector lengths in the order of 108. This improvement is comprised of both algorithmic changes (65%) and parallel-processing optimization (35%). The effectiveness of the new butterfly all-reduce is demonstrated using real-world neural network applications with the Spark framework. For the model-update operation we observe significant speed-ups using the new butterfly algorithm compared with the original reduce-broadcast, for both smaller (Cifar and Mnist) and larger (ImageNet) datasets.","PeriodicalId":298024,"journal":{"name":"Proceedings of the Machine Learning on HPC Environments","volume":"70 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123463619","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 8
BlazingText: Scaling and Accelerating Word2Vec using Multiple GPUs 使用多个gpu缩放和加速Word2Vec
Proceedings of the Machine Learning on HPC Environments Pub Date : 2017-11-12 DOI: 10.1145/3146347.3146354
Saurabh Gupta, Vineet Khare
{"title":"BlazingText: Scaling and Accelerating Word2Vec using Multiple GPUs","authors":"Saurabh Gupta, Vineet Khare","doi":"10.1145/3146347.3146354","DOIUrl":"https://doi.org/10.1145/3146347.3146354","url":null,"abstract":"Word2Vec is a popular algorithm used for generating dense vector representations of words in large corpora using unsupervised learning. The resulting vectors have been shown to capture semantic relationships between the corresponding words and are used extensively for many downstream natural language processing (NLP) tasks like sentiment analysis, named entity recognition and machine translation. Most open-source implementations of the algorithm have been parallelized for multi-core CPU architectures including the original C implementation by Mikolov et al. [1] and FastText [2] by Facebook. A few other implementations have attempted to leverage GPU parallelization but at the cost of accuracy and scalability. In this work, we present BlazingText, a highly optimized implementation of word2vec in CUDA, that can leverage multiple GPUs for training. BlazingText can achieve a training speed of up to 43M words/sec on 8 GPUs, which is a 9x speedup over 8-threaded CPU implementations, with minimal effect on the quality of the embeddings.","PeriodicalId":298024,"journal":{"name":"Proceedings of the Machine Learning on HPC Environments","volume":"5 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114146268","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 16
Accelerating deep neural network learning for speech recognition on a cluster of GPUs 在gpu集群上加速语音识别的深度神经网络学习
Proceedings of the Machine Learning on HPC Environments Pub Date : 2017-11-12 DOI: 10.1145/3146347.3146351
Guojing Cong, Brian Kingsbury, Soumyadip Gosh, G. Saon, Fan Zhou
{"title":"Accelerating deep neural network learning for speech recognition on a cluster of GPUs","authors":"Guojing Cong, Brian Kingsbury, Soumyadip Gosh, G. Saon, Fan Zhou","doi":"10.1145/3146347.3146351","DOIUrl":"https://doi.org/10.1145/3146347.3146351","url":null,"abstract":"We train deep neural networks to solve the acoustic modeling problem for large-vocabulary continuous speech recognition. We employ distributed processing using a cluster of GPUs. On modern GPUs, the sequential implementation takes over a day to train, and efficient parallelization without losing accuracy is notoriously hard. We show that ASGD methods for parallelization are not efficient for this application. Even with 4 GPUs, the overhead is significant, and the accuracies achieved are poor. We adapt a P-learner K-step model averaging algorithm that with 4 GPUs achieves accuracies comparable to that achieved by the sequential implementation. We further introduce adaptive measures that make our parallel implementation scale to the full cluster of 20 GPUs. Ultimately our parallel implementation achieves better accuracies than the sequential implementation with a 6.1 times speedup.","PeriodicalId":298024,"journal":{"name":"Proceedings of the Machine Learning on HPC Environments","volume":"216 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-12","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"122605597","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 4
Designing a Synchronization-reducing Clustering Method on Manycores: Some Issues and Improvements 设计一种多核减少同步的聚类方法:一些问题和改进
Proceedings of the Machine Learning on HPC Environments Pub Date : 2017-11-01 DOI: 10.1145/3146347.3146357
Weijian Zheng, Fengguang Song, Lan Lin
{"title":"Designing a Synchronization-reducing Clustering Method on Manycores: Some Issues and Improvements","authors":"Weijian Zheng, Fengguang Song, Lan Lin","doi":"10.1145/3146347.3146357","DOIUrl":"https://doi.org/10.1145/3146347.3146357","url":null,"abstract":"The k-means clustering method is one of the most widely used techniques in big data analytics. In this paper, we explore the ideas of software blocking, asynchronous local optimizations, and heuristics of simulated annealing to improve the performance of k-means clustering. Like most of the machine learning methods, the performance of k-means clustering relies on two main factors: the computing speed (per iteration), and the convergence rate. A straightforward realization of the software-blocking synchronization-reducing clustering algorithm, however, sees sporadic slower convergence rate than the standard k-means algorithm. To tackle the issues, we design an annealing-enhanced algorithm, which introduces the heuristics of stop conditions and annealing steps to provide as good or better performance than the standard k-means algorithm. This new enhanced k-means clustering algorithm is able to offer the same clustering quality as the standard k-means. Experiments with real-world datasets show that the new parallel implementation is faster than the open source HPC library of Parallel K-Means Data Clustering (e.g., 19% faster on relatively large datasets with 32 CPU cores, and 11% faster on a large dataset with 1,024 CPU cores). Moreover, the extent to which the program performance improves is largely determined by the actual convergence rate of applying the algorithm to different datasets.","PeriodicalId":298024,"journal":{"name":"Proceedings of the Machine Learning on HPC Environments","volume":"111 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-11-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129415507","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 3
TensorQuant: A Simulation Toolbox for Deep Neural Network Quantization TensorQuant:深度神经网络量化的仿真工具箱
Proceedings of the Machine Learning on HPC Environments Pub Date : 2017-10-13 DOI: 10.1145/3146347.3146348
D. Loroch, F. Pfreundt, N. Wehn, J. Keuper
{"title":"TensorQuant: A Simulation Toolbox for Deep Neural Network Quantization","authors":"D. Loroch, F. Pfreundt, N. Wehn, J. Keuper","doi":"10.1145/3146347.3146348","DOIUrl":"https://doi.org/10.1145/3146347.3146348","url":null,"abstract":"Recent research implies that training and inference of deep neural networks (DNN) can be computed with low precision numerical representations of the training/test data, weights and gradients without a general loss in accuracy. The benefit of such compact representations is twofold: they allow a significant reduction of the communication bottleneck in distributed DNN training and faster neural network implementations on hardware accelerators like FPGAs. Several quantization methods have been proposed to map the original 32-bit floating point problem to low-bit representations. While most related publications validate the proposed approach on a single DNN topology, it appears to be evident, that the optimal choice of the quantization method and number of coding bits is topology dependent. To this end, there is no general theory available, which would allow users to derive the optimal quantization during the design of a DNN topology. In this paper, we present a quantization tool box for the TensorFlow framework. TensorQuant allows a transparent quantization simulation of existing DNN topologies during training and inference. TensorQuant supports generic quantization methods and allows experimental evaluation of the impact of the quantization on single layers as well as on the full topology. In a first series of experiments with TensorQuant, we show an analysis of fix-point quantizations of popular CNN topologies.","PeriodicalId":298024,"journal":{"name":"Proceedings of the Machine Learning on HPC Environments","volume":"107 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2017-10-13","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123110010","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
引用次数: 20
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
相关产品
×
本文献相关产品
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:604180095
Book学术官方微信
小红书