Omnidirectional image quality assessment with gated dual-projection fusion

Abstract

Existing omnidirectional image quality assessment (OIQA) models typically rely on the equirectangular projection (ERP) or cubemap projection (CMP) of omnidirectional images as inputs. However, the deformation in ERP and the discontinuities at the boundaries of CMP limit the network’s ability to represent image information, leading to information loss. Therefore, it is necessary to fuse these two projections of omnidirectional images to achieve comprehensive feature representation. Current OIQA models only integrate and interact high-level features extracted from different projection formats at the last stage of the network, overlooking potential information loss at each stage within the network. To this end, we consider the respective strengths and weaknesses of the two projections, and design a feature extraction and fusion module at each stage of the network to enhance the model’s representation capability. Specifically, the ERP features are first decomposed into two projection formats before being fed into each feature extraction stage of the network for separate processing. Subsequently, we introduce the gating mechanism and develop a Gated Dual-Projection Fusion module (GDPF) to interactively fuse the features computed from both the ERP and CMP projection formats. GDPF allows the developed model to enhance critical information while filtering out deformation and discontinuous information. The fused features are then input into the next stage, where the aforementioned operations are repeated. This process alleviates the issues of feature representation caused by deformation in ERP and discontinuities in CMP and the fused features are used for quality prediction. Experiments on three public datasets demonstrate the superior prediction accuracy of the proposed model.