Siniša Veseli, John Hammonds, Steven Henke, Hannah Parraga, Barbara Frosik, Nicholas Schwarz
{"title":"用于实时数据处理的PvaPy流框架。","authors":"Siniša Veseli, John Hammonds, Steven Henke, Hannah Parraga, Barbara Frosik, Nicholas Schwarz","doi":"10.1107/S1600577525002115","DOIUrl":null,"url":null,"abstract":"<p><p>User facility upgrades, new measurement techniques, advances in data analysis algorithms as well as advances in detector capabilities result in an increasing amount of data collected at X-ray beamlines. Some of these data must be analyzed and reconstructed on demand to help execute experiments dynamically and modify them in real time. In turn, this requires a computing framework for real-time processing capable of moving data quickly from the detector to local or remote computing resources, processing data, and returning results to users. In this paper, we discuss the streaming framework built on top of PvaPy, a Python API for the EPICS pvAccess protocol. We describe the framework architecture and capabilities, and discuss scientific use cases and applications that benefit from streaming workflows implemented on top of this framework. We also illustrate the framework's performance in terms of achievable data-processing rates for various detector image sizes.</p>","PeriodicalId":48729,"journal":{"name":"Journal of Synchrotron Radiation","volume":"32 Pt 3","pages":"823-836"},"PeriodicalIF":2.5000,"publicationDate":"2025-05-01","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12067317/pdf/","citationCount":"0","resultStr":"{\"title\":\"PvaPy streaming framework for real-time data processing.\",\"authors\":\"Siniša Veseli, John Hammonds, Steven Henke, Hannah Parraga, Barbara Frosik, Nicholas Schwarz\",\"doi\":\"10.1107/S1600577525002115\",\"DOIUrl\":null,\"url\":null,\"abstract\":\"<p><p>User facility upgrades, new measurement techniques, advances in data analysis algorithms as well as advances in detector capabilities result in an increasing amount of data collected at X-ray beamlines. Some of these data must be analyzed and reconstructed on demand to help execute experiments dynamically and modify them in real time. In turn, this requires a computing framework for real-time processing capable of moving data quickly from the detector to local or remote computing resources, processing data, and returning results to users. In this paper, we discuss the streaming framework built on top of PvaPy, a Python API for the EPICS pvAccess protocol. We describe the framework architecture and capabilities, and discuss scientific use cases and applications that benefit from streaming workflows implemented on top of this framework. We also illustrate the framework's performance in terms of achievable data-processing rates for various detector image sizes.</p>\",\"PeriodicalId\":48729,\"journal\":{\"name\":\"Journal of Synchrotron Radiation\",\"volume\":\"32 Pt 3\",\"pages\":\"823-836\"},\"PeriodicalIF\":2.5000,\"publicationDate\":\"2025-05-01\",\"publicationTypes\":\"Journal Article\",\"fieldsOfStudy\":null,\"isOpenAccess\":false,\"openAccessPdf\":\"https://www.ncbi.nlm.nih.gov/pmc/articles/PMC12067317/pdf/\",\"citationCount\":\"0\",\"resultStr\":null,\"platform\":\"Semanticscholar\",\"paperid\":null,\"PeriodicalName\":\"Journal of Synchrotron Radiation\",\"FirstCategoryId\":\"101\",\"ListUrlMain\":\"https://doi.org/10.1107/S1600577525002115\",\"RegionNum\":3,\"RegionCategory\":\"物理与天体物理\",\"ArticlePicture\":[],\"TitleCN\":null,\"AbstractTextCN\":null,\"PMCID\":null,\"EPubDate\":\"2025/4/25 0:00:00\",\"PubModel\":\"Epub\",\"JCR\":\"\",\"JCRName\":\"\",\"Score\":null,\"Total\":0}","platform":"Semanticscholar","paperid":null,"PeriodicalName":"Journal of Synchrotron Radiation","FirstCategoryId":"101","ListUrlMain":"https://doi.org/10.1107/S1600577525002115","RegionNum":3,"RegionCategory":"物理与天体物理","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"2025/4/25 0:00:00","PubModel":"Epub","JCR":"","JCRName":"","Score":null,"Total":0}
PvaPy streaming framework for real-time data processing.
User facility upgrades, new measurement techniques, advances in data analysis algorithms as well as advances in detector capabilities result in an increasing amount of data collected at X-ray beamlines. Some of these data must be analyzed and reconstructed on demand to help execute experiments dynamically and modify them in real time. In turn, this requires a computing framework for real-time processing capable of moving data quickly from the detector to local or remote computing resources, processing data, and returning results to users. In this paper, we discuss the streaming framework built on top of PvaPy, a Python API for the EPICS pvAccess protocol. We describe the framework architecture and capabilities, and discuss scientific use cases and applications that benefit from streaming workflows implemented on top of this framework. We also illustrate the framework's performance in terms of achievable data-processing rates for various detector image sizes.
期刊介绍:
Synchrotron radiation research is rapidly expanding with many new sources of radiation being created globally. Synchrotron radiation plays a leading role in pure science and in emerging technologies. The Journal of Synchrotron Radiation provides comprehensive coverage of the entire field of synchrotron radiation and free-electron laser research including instrumentation, theory, computing and scientific applications in areas such as biology, nanoscience and materials science. Rapid publication ensures an up-to-date information resource for scientists and engineers in the field.