Network-aware Data Management最新文献

Design and implementation of control sequence generator for SDN-enhanced MPI 为 SDN 增强型 MPI 设计和实施控制序列生成器

Network-aware Data Management Pub Date : 2015-11-15 DOI: 10.1145/2832099.2832103

Baatarsuren Munkhdorj, Keichi Takahashi, Khureltulga Dashdavaa, Yasuhiro Watashiba, Y. Kido, S. Date, S. Shimojo

{"title":"Design and implementation of control sequence generator for SDN-enhanced MPI","authors":"Baatarsuren Munkhdorj, Keichi Takahashi, Khureltulga Dashdavaa, Yasuhiro Watashiba, Y. Kido, S. Date, S. Shimojo","doi":"10.1145/2832099.2832103","DOIUrl":"https://doi.org/10.1145/2832099.2832103","url":null,"abstract":"MPI (Message Passing Interface) offers a suite of APIs for inter-process communication among parallel processes. We have approached to the acceleration of MPI collective communication such as MPI_Bcast and MPI_Allreduce, taking advantage of network programmability brought by Software Defined Networking (SDN). The basic idea is to allow a SDN controller to dynamically control the packet flows generated by MPI collective communication based on the communication pattern and the underlying network conditions. Although our research have succeeded to accelerate an MPI collective communication in terms of execution time, the switching of network control functionality for MPI collective communication along MPI program execution have not been considered yet. This paper presents a mechanism that provides the control sequence for SDN controller to control packet flows based on the communication plan for the entire MPI application. The control sequence encloses a chronologically ordered list of the MPI collectives operated in the MPI application and the process-related information of each in the list. To verify if the SDN-enhanced MPI collectives can be used in combination with the proposed mechanism, the envisioned environment was prototyped. As a result, SDN-enhanced MPI collectives were able to be used in combination.","PeriodicalId":108576,"journal":{"name":"Network-aware Data Management","volume":"37 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2015-11-15","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128519033","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 1

Hysteresis-based optimization of data transfer throughput 基于迟滞的数据传输吞吐量优化

Network-aware Data Management Pub Date : 2015-11-15 DOI: 10.1145/2832099.2832104

M. S. Q. Z. Nine, Kemal Guner, T. Kosar

引用次数: 14

Managing scientific data with named data networking 使用命名数据网络管理科学数据

Network-aware Data Management Pub Date : 2015-11-15 DOI: 10.1145/2832099.2832100

Chengyu Fan, Susmit Shannigrahi, S. DiBenedetto, C. Olschanowsky, C. Papadopoulos, H. Newman

引用次数: 39

A multi-domain SDN for dynamic layer-2 path service 多域SDN，用于动态第二层路径服务

Network-aware Data Management Pub Date : 2015-11-15 DOI: 10.1145/2832099.2832101

S. Tepsuporn, Fatma Alali, M. Veeraraghavan, Xiang Ji, Brian Cashman, Andrew J. Ragusa, Luke Fowler, C. Guok, T. Lehman, Xi Yang

引用次数: 7

Approximate causal consistency for partially replicated geo-replicated cloud storage 部分复制的地理复制云存储的近似因果一致性

Network-aware Data Management Pub Date : 2015-11-15 DOI: 10.1145/2832099.2832102

A. Kshemkalyani, T. Hsu

引用次数: 5

Network-aware virtual machine consolidation for large data centers 面向大型数据中心的网络感知虚拟机整合

Network-aware Data Management Pub Date : 2013-11-17 DOI: 10.1145/2534695.2534702

Dharmesh Kakadia, N. Kopri, Vasudeva Varma

{"title":"Network-aware virtual machine consolidation for large data centers","authors":"Dharmesh Kakadia, N. Kopri, Vasudeva Varma","doi":"10.1145/2534695.2534702","DOIUrl":"https://doi.org/10.1145/2534695.2534702","url":null,"abstract":"Resource management in modern data centers has become a challenging task due to the tremendous growth of data centers. In large virtual data centers, performance of applications is highly dependent on the communication bandwidth available among virtual machines. Traditional algorithms either do not consider network I/O details of the applications or are computationally intensive. We address the problem of identifying the virtual machine clusters based on the network traffic and placing them intelligently in order to improve the application performance and optimize the network usage in large data center. We propose a greedy consolidation algorithm that ensures the number of migrations is small and the placement decisions are fast, which makes it practical for large data centers. We evaluated our approach on real world traces from private and academic data centers, using simulation and compared the existing algorithms on various parameters like scheduling time, performance improvement and number of migrations. We observed a ~70% savings of the interconnect bandwidth and overall ~60% improvements in the applications performances. Also, these improvements were produced within a fraction of scheduling time and number of migrations.","PeriodicalId":108576,"journal":{"name":"Network-aware Data Management","volume":"106 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132400998","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 34

End-to-end data movement using MPI-IO over routed terabits infrastructures 在路由太比特基础设施上使用MPI-IO的端到端数据移动

Network-aware Data Management Pub Date : 2013-11-17 DOI: 10.1145/2534695.2534705

Geoffroy R. Vallée, S. Atchley, Youngjae Kim, G. Shipman

{"title":"End-to-end data movement using MPI-IO over routed terabits infrastructures","authors":"Geoffroy R. Vallée, S. Atchley, Youngjae Kim, G. Shipman","doi":"10.1145/2534695.2534705","DOIUrl":"https://doi.org/10.1145/2534695.2534705","url":null,"abstract":"Scientific discovery is nowadays driven by large-scale simulations running on massively parallel high-performance computing (HPC) systems. These applications each generate a large amount of data, which then needs to be post-processed for example for data mining or visualization. Unfortunately, the computing platform used for post processing might be different from the one on which the data is initially generated, introducing the challenge of moving large amount of data between computing platforms. This is especially challenging when these two platforms are geographically separated since the data needs to be moved between computing facilities. This is even more critical when scientists tightly couple their domain specific applications with a post processing application.\u0000 The paper presents a solution for the data transfer between MPI applications using a dedicated wide area network (WAN) terabit infrastructure. The proposed solution is based on parallel access to data files and the Message Passing Interface (MPI) over the Common Communication Infrastructure (CCI) for the data transfer over a routed infrastructure. In the context of this research, the Energy Sciences Network (ESnet) of the U.S. Department of Energy (DOE) is targeted for the transfer of data between DOE national laboratories.","PeriodicalId":108576,"journal":{"name":"Network-aware Data Management","volume":"44 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"131576135","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 3

In-network, push-based network resource monitoring: scalable, responsive network management 网络内，基于推送的网络资源监控:可扩展，响应式网络管理

Network-aware Data Management Pub Date : 2013-11-17 DOI: 10.1145/2534695.2534704

Taylor L. Groves, D. Arnold, Yihua He

{"title":"In-network, push-based network resource monitoring: scalable, responsive network management","authors":"Taylor L. Groves, D. Arnold, Yihua He","doi":"10.1145/2534695.2534704","DOIUrl":"https://doi.org/10.1145/2534695.2534704","url":null,"abstract":"We present preliminary work from our experiences with distributed, push-based monitoring of networks at Yahoo!. Network switches have grown beyond mere ASICs into machines which support unmodified Linux kernels and familiar user interfaces. These advances have enabled a paradigm shift in network monitoring. In lieu of traditional approaches where network diagnostics were delivered via SNMP we utilize Sysdb of Arista's EOS to implement a push based approach to network monitoring. This leaves the individual switches in charge of determining what monitoring data to send and when to send it. With this approach -- on-switch collection, dissemination, and analysis of interfaces and protocols become possible. This push based approach reduces the feedback loop of network diagnostics and enables networkaware applications, middleware and resource managers to have access to the freshest available data.\u0000 Our work utilizes the OpenTSDB monitoring framework to provide a scalable back-end for accessing and storing real-time statistics delivered by on-switch collection agents. OpenTSDB is built on top of Hadoop/HBase, which handles the underlying access and storage for the monitoring system. We wrote two collection agents as prototypes to explore the framework and demonstrate the benefits of push based network monitoring.","PeriodicalId":108576,"journal":{"name":"Network-aware Data Management","volume":"21 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128573280","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 4

Efficient wide area data transfer protocols for 100 Gbps networks and beyond 适用于100gbps及以上网络的高效广域数据传输协议

Network-aware Data Management Pub Date : 2013-11-17 DOI: 10.1145/2534695.2534699

E. Kissel, D. M. Swany, B. Tierney, Eric Pouyoul

引用次数: 19

Characterizing the impact of end-system affinities on the end-to-end performance of high-speed flows 表征端系统亲和关系对高速流动端到端性能的影响

Network-aware Data Management Pub Date : 2013-11-17 DOI: 10.1145/2534695.2534697

Nathan Hanford, V. Ahuja, Mehmet Balman, M. Farrens, D. Ghosal, Eric Pouyoul, B. Tierney

{"title":"Characterizing the impact of end-system affinities on the end-to-end performance of high-speed flows","authors":"Nathan Hanford, V. Ahuja, Mehmet Balman, M. Farrens, D. Ghosal, Eric Pouyoul, B. Tierney","doi":"10.1145/2534695.2534697","DOIUrl":"https://doi.org/10.1145/2534695.2534697","url":null,"abstract":"Multi-core end-systems use Receive Side Scaling (RSS) to parallelize protocol processing. RSS uses a hash function on the standard flow descriptors and an indirection table to assign incoming packets to receive queues which are pinned to specific cores. This ensures flow affinity in that the interrupt processing of all packets belonging to a specific flow is processed by the same core. A key limitation of standard RSS is that it does not consider the application process that consumes the incoming data in determining the flow affinity. In this paper, we carry out a detailed experimental analysis of the performance impact of the application affinity in a 40 Gbps testbed network with a dual hexa-core end-system. We show, contrary to conventional wisdom, that when the application process and the flow are affinitized to the same core, the performance (measured in terms of end-to-end TCP throughput) is significantly lower than the line rate. Near line rate performance is observed when the flow and the application process are affinitized to different cores belonging to the same socket. Furthermore, affinitizing the application and the flow to cores on different sockets results in significantly lower throughput than the line rate. These results arise due to the memory bottleneck, which is demonstrated using preliminary correlational data on the cache hit rate in the core that services the application process.","PeriodicalId":108576,"journal":{"name":"Network-aware Data Management","volume":"63 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2013-11-17","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115925305","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}

引用次数: 16