{"title":"In situ particle advection via parallelizing over particles","authors":"R. Binyahib, D. Pugmire, H. Childs","doi":"10.1145/3364228.3364235","DOIUrl":"https://doi.org/10.1145/3364228.3364235","url":null,"abstract":"We extend the method for particle advection that parallelizes over particles to work in an in situ setting. We then compare our method with the typical method for in situ, parallelizing over data. Our experiments consist of parallelism at 512 cores, a data set with 67 million cells, and ten billion total advection steps. Our findings show that parallelizing over particles can be more than ten times faster for some workloads, for reasonable memory cost. Overall, the significance of these findings is to demonstrate that moving data can be worthwhile in some in situ settings.","PeriodicalId":400017,"journal":{"name":"Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization","volume":"5 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-11-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"125404368","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Matthieu Dorier, Orcun Yildiz, T. Peterka, R. Ross
{"title":"The challenges of elastic in situ analysis and visualization","authors":"Matthieu Dorier, Orcun Yildiz, T. Peterka, R. Ross","doi":"10.1145/3364228.3364234","DOIUrl":"https://doi.org/10.1145/3364228.3364234","url":null,"abstract":"In situ analysis and visualization have been proposed in high-performance computing (HPC) to enable executing analysis tasks while a simulation is running, bypassing the parallel file system and avoiding the need for storing massive amounts of data. One aspect of in situ analysis that has not been extensively researched to date, however, is elasticity. Current in situ analysis frameworks use a fixed amount of resources and can hardly be scaled up or down dynamically throughout the simulation's run time as a response to changes in the requirements. In this paper, we present the challenges posed by elastic in situ analysis and visualization. We emphasize that elasticity can take various forms. We show the difficulties of supporting each form of elasticity with the state-of-the-art HPC technologies, and we suggest solutions to overcome these difficulties. The resulting four-way classification can be seen as a taxonomy for future elastic in situ analysis and visualization systems.","PeriodicalId":400017,"journal":{"name":"Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization","volume":"1 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-11-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132156258","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Tobias Rau, P. Gralka, Oliver Fernandes, G. Reina, S. Frey, T. Ertl
{"title":"The impact of work distribution on in situ visualization: a case study","authors":"Tobias Rau, P. Gralka, Oliver Fernandes, G. Reina, S. Frey, T. Ertl","doi":"10.1145/3364228.3364233","DOIUrl":"https://doi.org/10.1145/3364228.3364233","url":null,"abstract":"Large-scale computer simulations generate data at rates that necessitate visual analysis tools to run in situ. The distribution of work on and across nodes of a supercomputer is crucial to utilize compute resources as efficiently as possible. In this paper, we study two work distribution problems in the context of in situ visualization and jointly assess the performance impact of different variants. First, especially for simulations involving heterogeneous loads across their domain, dynamic load balancing can significantly reduce simulation run times. However, the adjustment of the domain partitioning associated with this also has a direct impact on visualization performance. The exact impact of this side effect is largely unclear a priori as generally different criteria are used for balancing simulation and visualization load. Second, on node level, the adequate allocation of threads to simulation or visualization tasks minimizes the performance drain of the simulation while also enabling timely visualization results. In our case study, we jointly study both work distribution aspects with the visualization framework MegaMol coupled in situ on node level to the molecular dynamics simulation ls1 Mardyn on Stampede2 at TACC.","PeriodicalId":400017,"journal":{"name":"Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization","volume":"52 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-11-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115304902","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"A new approach for in situ analysis in fully coupled earth system models","authors":"U. Turuncoglu, Baris Önol, M. Ilıcak","doi":"10.1145/3364228.3364231","DOIUrl":"https://doi.org/10.1145/3364228.3364231","url":null,"abstract":"This work focuses on enabling the use of existing in situ analysis and visualization methods within the model coupling framework, which is specially designed software layer to combine physical components of Earth system (i.e., atmosphere, ocean) through a standardized calling interface to simplify common tasks such as interpolation among different numerical grids and synchronization of the model components. The new approach enables simultaneous analysis of the vast amount of data produced by a multi-component Earth system model in an integrated way and facilitates efficient, interoperable, and optimized data analysis and visualization workflows for Earth system science applications. To demonstrate the flexibility and the capability of the new approach, a fully coupled regional atmosphere-ocean modeling application that aims to investigate air-sea interaction in a very high spatial and temporal resolution has been implemented. In the use case application, both the simulation and in situ visualization application run in parallel on a large-scale HPC platform with both software and hardware rendering support through the use of accelerators.","PeriodicalId":400017,"journal":{"name":"Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization","volume":"27 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-11-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"134181224","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"HDF5 as a vehicle for in transit data movement","authors":"Junmin Gu, B. Loring, Kesheng Wu, E. W. Bethel","doi":"10.1145/3364228.3364237","DOIUrl":"https://doi.org/10.1145/3364228.3364237","url":null,"abstract":"For in transit processing, one of the fundamental challenges is the efficient movement of data from producers to consumers. Exploiting the flexibility offered by the SENSEI generic in situ framework, we have developed a number of different in transit data transport mechanisms. In this work, we focus on the transport mechanism that leverages the HDF5 parallel I/O library, and investigate the performance characteristics of this transport mechanism. For in transit use cases at scale on HPC platforms, one might expect that an in transit data transport mechanism that uses faster layers of the storage hierarchy, such as DRAM memory, would always outperform a transport that uses slower layers of the storage hierarchy, such as an NVRAM-based persistent storage presented as a distributed file system. However, our test results show that the performance of the transport using NVRAM is competitive with the transport that uses socket-based data movement across varying levels of producer and consumer concurrency.","PeriodicalId":400017,"journal":{"name":"Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization","volume":"27 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-11-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"128882598","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Yoshiaki Yamaoka, Kengo Hayashi, Naohisa Sakamoto, J. Nonaka
{"title":"In situ adaptive timestep control and visualization based on the spatio-temporal variations of the simulation results","authors":"Yoshiaki Yamaoka, Kengo Hayashi, Naohisa Sakamoto, J. Nonaka","doi":"10.1145/3364228.3364230","DOIUrl":"https://doi.org/10.1145/3364228.3364230","url":null,"abstract":"Effective visualization of time-varying volumetric data is considered challenging, since in addition to the traditional visualization parameter selections, there is also the need to take care about the dynamic changings between the timesteps. In situ adaptive sampling based on the amount of change between the simulation timepsteps might be helpful for the I/O and visualization cost savings. In this paper, we propose the use of Kernel Density Estimation (KDE) and Kullback-Leibler (KL) divergence, in order to estimate the amount of internal variations between the timesteps, and use this as a parameter for controlling the sampling rate. Needless to say that these selected timesteps should be sufficient to enable the understanding of the underlying phenomena without missing important data features. We confirmed the effectiveness on reducing the number of timepsteps, by using an OpenFOAM CFD simulation with irregular volume data. However, we could also observe that the user-defined parameter selection can highly influence the amount of data size reduction as well as the smoothness of the visualization results, and an optimal parameter selection method remains as a future work.","PeriodicalId":400017,"journal":{"name":"Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization","volume":"20 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-11-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"114461021","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Seif Ibrahim, T. Stitt, Matthew Larsen, C. Harrison
{"title":"Interactive in situ visualization and analysis using Ascent and Jupyter","authors":"Seif Ibrahim, T. Stitt, Matthew Larsen, C. Harrison","doi":"10.1145/3364228.3364232","DOIUrl":"https://doi.org/10.1145/3364228.3364232","url":null,"abstract":"There is a gap in interactive in situ solutions for HPC simulations. While we have access to fully-featured visualization UIs through tools like ParaViewCatalyst and VisItLibSim,we lack in situ infrastructure to access more general interactive environments like Jupyter. The Jupyter ecosystem of tools provides a rich paradigm for interactive data analysis and is well-suited to help expand interactive use of in situ. Both the complexity of the Jupyter ecosystem and the security requirements of HPC centers pose challenges to developing software infrastructure that enables direct use of Jupyter in simulation codes. With this work, we developed a system enabling simulations instrumented with Ascent to connect to Jupyter. This system allows simulation users to interact with their data in situ using Jupyter Notebooks. The system combines Ascent's embedded Python filter infrastructure with a Client/Server Jupyter Bridge Kernel to simplify both deployment and security considerations on HPC systems. We describe the design of this system, demonstrate basic usage, and describe a prototype of an Ascent rendering UI built on top of this system.","PeriodicalId":400017,"journal":{"name":"Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization","volume":"22 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-11-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"115281703","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization","authors":"","doi":"10.1145/3364228","DOIUrl":"https://doi.org/10.1145/3364228","url":null,"abstract":"","PeriodicalId":400017,"journal":{"name":"Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization","volume":"19 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-11-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"133375515","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jeff Mauldin, Thomas J. Otahal, A. Agelastos, S. Domino
{"title":"In-situ visualization for the large scale computing initiative milestone","authors":"Jeff Mauldin, Thomas J. Otahal, A. Agelastos, S. Domino","doi":"10.1145/3364228.3364229","DOIUrl":"https://doi.org/10.1145/3364228.3364229","url":null,"abstract":"The Sandia National Laboratories (SNL) Large-Scale Computing Initiative (LSCI) milestone required running two parallel simulation codes at scale on the Trinity supercomputer at Los Alamos National Laboratory (LANL) to obtain presentation quality visualization results via in-situ methods. The two simulation codes used were Sandia Parallel Aerosciences Research Code (SPARC) and Nalu, both fluid dynamics codes developed at SNL. The codes were integrated with the ParaView Catalyst in-situ visualization library via the SNL developed Input Output SubSystem (IOSS). The LSCI milestone had a relatively short time-scale for completion of two months. During setup and execution of in-situ visualization for the milestone, there were several challenging issues in the areas of software builds, parallel startup-times, and in the a priori specification of visualizations. This paper will discuss the milestone activities and technical challenges encountered in its completion.","PeriodicalId":400017,"journal":{"name":"Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization","volume":"24 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-11-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"123596985","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Sergei Shudler, N. Ferrier, J. Insley, M. Papka, S. Rizzi
{"title":"Spack meets singularity: creating movable in-situ analysis stacks with ease","authors":"Sergei Shudler, N. Ferrier, J. Insley, M. Papka, S. Rizzi","doi":"10.1145/3364228.3364682","DOIUrl":"https://doi.org/10.1145/3364228.3364682","url":null,"abstract":"In-situ data analysis and visualization is a promising technique to handle the enormous amount of data an extreme-scale application produces. One challenge users often face in adopting in-situ techniques is setting the right environment on a target machine. Platforms such as SENSEI require complex software stacks that consist of various analysis packages and visualization applications. The user has to make sure all these prerequisites exist on the target machine, which often involves compiling and setting them up from scratch. In this paper, we leverage the containers technology (e.g., light-weight virtualization images) and provide users with Singularity containers that encapsulate ready-to-use, movable in-situ software stacks. Moreover, we make use of Spack to ease the process of creating these containers. Finally, we evaluate this solution by running in-situ analysis from within a container on an HPC system.","PeriodicalId":400017,"journal":{"name":"Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization","volume":"61 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2019-11-18","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"132052446","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}