Low-frequency Radio Interferometric Array Imaging Pipeline Optimization Based on Distributed Execution Framework

Abstract

The Square Kilometre Array (SKA) project is an international collaboration to build the world’s largest radio telescope, whose sensitivity and measurement speed will be an order of magnitude higher than those of all current radio telescopes. Radio continuum survey is one of the main observation mode of the SKA, and the establishment of a standard map of the survey area based on continuum imaging will provide an important foundation for subsequent astronomical science. The GaLactic and Extragalactic All-sky Murchison Widefield Array survey eXtended (GLEAM-X) is a project of the SKA pilot telescope Murchison Widefield Array (MWA) in 2018—2020. GLEAM-X is a new radio continuum survey project to be carried out with the MWA Phase II expansion array in 2018—2020. The experience of optimizing the imaging pipeline based on the distributed execution framework will help to solve the problem of massive data processing. In this paper, we describe the process steps of GLEAM-X imaging pipeline, integrate and improve it, and realize parallel processing of multiple pipelines on the China SKA Regional Centre Prototype (CSRC-P), and verify the deployment and test the correctness of the imaging pipeline system using GLEAM-X observation data. The GLEAM-X observations were used to validate the deployment of the imaging pipeline system and test its correctness. Then, to optimize the pipelines and improve the processing efficiency, the Data Activated Liu Graph Engine (DALiuGE) was used to integrate the imaging pipelines into the DALiuGE execution framework to automate the distributed parallel processing of the pipelines. Performance tests and results’ analysis show that the optimized imaging pipeline based on the DALiuGE execution framework has better performance, more flexible adaptability, and scalability than the traditional parallel approach, and can support future large-scale continuum imaging experiments during the first phase of SKA commissioning.