Single sensor multi-spectral imaging

Abstract

This dissertation presents the benefits of using a multispectral Single Sensor Camera (SSC) that, simultaneously acquire images in the visible and near-infrared (NIR) bands. The principal benefits while addressing problems related to image bands in the spectral range of 400 to 1100 nanometers, there are cost reductions in the hardware setup because only one SSC is needed instead of two; moreover, the cameras’ calibration and images alignment are not required anymore. Concerning to the NIR spectrum, even though this band is close to the visible band and shares many properties, the sensor sensitivity is material dependent due to different behavior of absorption/reflectance capturing a given scene compared to visible channels. Many works in literature have proven the benefits of working with NIR to enhance RGB images (e.g., image enhancement, dehazing, etc.). In spite of the advantage of using SSC (e.g., low latency), there are some drawbacks to be solved. One of these drawbacks corresponds to the nature of the silicon-based sensor, which in addition to capturing the RGB image when the infrared cut off filter is not installed it also acquires NIR information into the visible image. This phenomenon is called RGB and NIR crosstalking. This thesis firstly faces this problem in challenging images and then it shows the benefit of using multispectral images in the edge detection task. Then, three methods based on CNN have been proposed for edge detection. While the first one is based on the most used model, holistically-nested edge detection (HED) termed as multispectral HED (MS-HED), the other two have been proposed observing the drawbacks of MS-HED. These two novel architectures have been designed from scratch; after the first architecture is validated in the visible domain a slight redesign is proposed to tackle the multispectral domain. A dataset is collected to face this problem with SSCs. Even though edge detection is confronted in the multispectral domain, its qualitative and quantitative evaluation demonstrates the generalization in other datasets used for edge detection, improving state-of-the-art results. One of the main properties of this proposal is to show that the edge detection problem can be tackled by just training the proposed architecture one-time while validating it in other datasets.