Registration of close-range, multi-lens multispectral imagery by retrieving the scene 3D structure

Abstract

Multispectral, multi-lens cameras, which acquire spectal images from different individual cameras equipped with different optical filters, are among the most widely used multispectral cameras available on the market. However, their use for close-range sensing is limited by the lack of registration algorithms capable of handling the strong parallax effects observed on scenes with non-negligible relief. In this paper, we propose a method based on stereo camera calibration and disparity estimation to register a close-range multispectral image while retrieving the corresponding 3D point cloud. The method takes advantage of the rigidity of these cameras and the synchronized capture of multispectral bands, both of which are thus compulsory. The algorithm is three-fold. First, the optimal combination of band pair alignments is found. Then, the semi-global matching stereovision algorithm combined with a robust matching cost function are used to align these band pairs and to compute the point cloud. Finally, a pixel filling step that exploits the spectral covariances of the different classes of materials in the image is implemented to limit the number of missing pixels, e.g., due to occlusions.

The method was tested on Airphen multispectral images of four plant crops (wheat, sunflower, cover crops and maize) acquired at a distance to the ground ranging from 1.5 to 3 m, thus encompassing a large variability in 3D structure and parallax effects. The results demonstrate that the proposed method achieves better registration performance than six state-of-the-art existing methods, while maintaining a reasonable processing time. Further, the point cloud provides accurate information on the 3D structure of the imaged scene, as shown by the centimetric plant height estimation accuracy. As the point cloud is aligned with the registered multispectral bands, the method provides a 4D (spectral and spatial) description of the scene with a single image, i.e., a multispectral point cloud. This opens up interesting prospects for several applications in close-range sensing including, but not restricted to, vegetation characterization.