Observability driven Multi-modal Line-scan Camera Calibration

Multi-modal sensors such as hyperspectral line-scan and frame cameras can be incorporated into a single camera system, enabling individual sensor limitations to be compensated. Calibration of such systems is crucial to ensure data from one modality can be related to the other. The best known approach is to capture multiple measurements of a known planar pattern, which are then used to optimize calibration parameters through non-linear least squares. The confidence in the optimized parameters is dependent on the measurements, which are contaminated by noise due to sensor hardware. Understanding how this noise transfers through the calibration is essential, especially when dealing with line-scan cameras that rely on measurements to extract feature points. This paper adopts a maximum likelihood estimation method for propagating measurement noise through the calibration, such that the optimized parameters are associated with an estimate of uncertainty. The uncertainty enables development of an active calibration algorithm, which uses observability to selectively choose images that improve parameter estimation. The algorithm is tested in both simulation and hardware, then compared to a naive approach that uses all images to calibrate. The simulation results for the algorithm show a drop of 26.4% in the total normalized error and 46.8% in the covariance trace. Results from the hardware experiments also show a decrease in the covariance trace, demonstrating the importance of selecting good measurements for parameter estimation.