On-the-Fly Data Analysis and Reduction

X-ray facilities are currently going through a wave of upgrades. In the synchrotron world, upgrading to the 4th generation machine is the goal for many facilities. MAX IV (Sweden), ESRF-EBS (France), and Sirius (Brazil) are the pioneers; APS (USA) just started its upgrade; HEPS (China) is under construction; and SLS 2.0 upgrade (Switzerland) will begin shortly. For X-ray free electron lasers, LCLS-II (USA) will be capable of producing 1 million pulses per second, a fourorder-of-magnitude improvement over LCLS in repetition rate. Beamlines will also receive enhancements together with accelerators to enable groundbreaking research. The difficulties posed by growing data volumes and the requirements for computational power, however, are shared by all of these upgrade programs. For instance, upgraded facilities could generate tens of petabytes of data annually, and kilohertz realtime data analysis is anticipated. In addition, the performance requirements of computing components like microprocessors, disks, and network interfaces are increasing more slowly than the IT infrastructure requirements of synchrotron facilities. It is unrealistic to believe that the anticipated increase in data creation can be accommodated by just waiting for the next chip generation. To cope with this pressing issue, X-ray facilities may have to make a difficult decision: either expand investment budget in IT infrastructure dramatically, or restrict experiment performance. Therefore, we believe innovative solutions are required to meet the challenge through altering community practice. The examples provided in this special edition of Synchrotron Radiation News illustrate how this is feasible. Nikitin et al. present a real-time processing solution for a 2-BM tomography beamline at the APS. Having instant feedback about the experiment’s progress allows the experiment team to collect only the necessary data. In addition, the software allows for focusing only on events of interest. The paper also gives insight into the algorithms and computing infrastructure essential for real-time operation, with part of the calculation accelerated on a GPU. Blanschke et al. present the advantages of using external high-performance computing facilities. The paper describes a fruitful collaboration between LCLS and the National Energy Research Scientific Computing Center (Berkeley, USA). Images collected at the Xray free electron laser are processed in realtime at the supercomputing center, providing quick feedback to the experiment team. By using external resources, the facility does not need to operate large clusters on-site, but can adjust the size and type of external computing resources based on the demand of particular experiments. Underwood et al. describe a lossy compression scheme for serial macromolecular crystallography for LCLS. The method is based on the observation that only a small fraction of pixels—those with and near Bragg spots—in a protein diffraction image contribute to solving a 3D structure. The authors of the paper show that the data quality will not degrade if the remaining background pixels are compressed with a lossy algorithm. By doing so, they can get even an order of magnitude better compression as compared to the current lossless approach. Finally, this issue includes a contribution by M. Burian et al. The article presents recent developments in streaming functionality from commercial detectors produced by DECTRIS. This option enables analysis and processing of X-ray images prior to saving them on disk. An important aspect of the functionality is including detector and beamline metadata with the streamed images. n