J. Choi, T. Kurç, Jeremy S. Logan, M. Wolf, E. Suchyta, James Kress, D. Pugmire, N. Podhorszki, Eun-Kyu Byun, M. Ainsworth, M. Parashar, S. Klasky
{"title":"Stream processing for near real-time scientific data analysis","authors":"J. Choi, T. Kurç, Jeremy S. Logan, M. Wolf, E. Suchyta, James Kress, D. Pugmire, N. Podhorszki, Eun-Kyu Byun, M. Ainsworth, M. Parashar, S. Klasky","doi":"10.1109/NYSDS.2016.7747804","DOIUrl":"https://doi.org/10.1109/NYSDS.2016.7747804","url":null,"abstract":"The demand for near real-time analysis of streaming data is increasing rapidly in scientific projects. This trend is driven by the fact that it is expensive and time consuming to design and execute complex experiments and simulations. During an experiment, the research team and the team at the experiment facility will want to analyze data as it is generated, interpret it, and collaboratively make decisions to modify the experiment parameters or abort the experiment in order to prevent events that may damage experimental instruments or to avoid wasting resources if there is a problem. The increasing velocity and volume of streaming data and the multi-institutional nature of large-scale scientific projects present challenges to near real-time analysis of streaming data. In this work we develop a framework to address these challenges. This framework provides an interface for applications to define and interact with named, self-describing streams, takes advantage of advanced network technologies, and implements support for the reduction and compression of data at the source. We describe this framework and demostrate its application in three scientific applications.","PeriodicalId":406964,"journal":{"name":"2016 New York Scientific Data Summit (NYSDS)","volume":"8 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"129637611","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Daniel Cisek, J. Dale, Susan Pepper, M. Mahajan, Shinjae Yoo
{"title":"Parking lot delineation and object detection using a localized Convolutional Neural Network","authors":"Daniel Cisek, J. Dale, Susan Pepper, M. Mahajan, Shinjae Yoo","doi":"10.1109/NYSDS.2016.7747821","DOIUrl":"https://doi.org/10.1109/NYSDS.2016.7747821","url":null,"abstract":"Satellite imagery analysis is becoming increasingly important in a variety of fields. Automation of analysts' workflows has the potential to greatly increase their efficiency and effectiveness. Specifically, developing automatic parking lot extraction from overhead imagery has huge importance to predicting stores future sales. However, to the best of our knowledge, there is no prior work for the development of an end-to-end pipeline for this task. This paper outlines a method for automating parking lot extraction from overhead imagery. We address this challenge using the combination of a Convolutional Neural Network (CNN) and various morphological tools of analysis. Our pilot end-to-end pipeline has been shown to be effective even in complex environments, and exhibits promising results for the automation of imagery analysis.","PeriodicalId":406964,"journal":{"name":"2016 New York Scientific Data Summit (NYSDS)","volume":"20 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"116218701","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Analyzing large data sets from XGC1 magnetic fusion simulations using apache spark","authors":"R. Michael Churchill","doi":"10.1109/NYSDS.2016.7747820","DOIUrl":"https://doi.org/10.1109/NYSDS.2016.7747820","url":null,"abstract":"Apache Spark is explored as a tool for analyzing large data sets from the magnetic fusion simulation code XGC1. Implementation details of Apache Spark on the NERSC Edison supercomputer are discussed, including binary file reading, and parameter setup. An unsupervised machine learning algorithm, k-means clustering, is applied to XGC1 particle distribution function data, showing that highly turbulent spatial regions do not have common coherent structures, but rather broad, ringlike structures in velocity space.","PeriodicalId":406964,"journal":{"name":"2016 New York Scientific Data Summit (NYSDS)","volume":"118 1","pages":"0"},"PeriodicalIF":0.0,"publicationDate":"2016-08-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"117283787","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":0,"RegionCategory":"","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}