3D Hyperspectral Light-Field Imaging: a first intraoperative implementation

Abstract Hyperspectral imaging is an emerging technology that has gained significant attention in the medical field due to its ability to provide precise and accurate imaging of biological tissues. The current methods of hyperspectral imaging, such as filter-wheel, snapshot, line-scanning, and push-broom cameras have limitations such as low spatial and spectral resolution, slow acquisition time. New developments on the field of light field cameras show the potential to overcome these limitations. In this paper, we use a novel hyperspectral lightfield camera and try to combine the capability of hyperspectral and 3D analysis. For this purpose we calibrate our system and test it during two ENT-surgeries to show its potential for improving surgical outcomes. The micro-lenses of the camera map 66 spectral sub-images onto the sensor allowing to reconstruct the spectral behavior of the captured scene in the spectral range of 350-1000nm. In addition, we use the sensor data to apply a 3D camera calibration pipeline to allow 3D surface reconstruction. We captured 26 calibration images and achieved calibration results in accordance to stated company data. The best calibration showed a re-projection error of 0.55 px. Further, we tested the camera during a parotidectomy and a neck-dissection. The extracted reflectance spectra of the selected venal and arterial regions correspond perfectly to the spectrum of oxygenated and deoxygenated hemoglobin. For the first time, up to our knowledge, a hyperspectral lightfield camera has been used during a surgery. We were able to continuously capture images and analyze the reconstructed spectra of specific tissue types. Further, we are able to use the sensor data of the micro-lens projections to calibrate the multilens camera system for later intraoperative measurement tasks.