{"title":"High Spatio-Temporal Imaging Reconstruction for Hybrid Spike-RGB Cameras","authors":"Lujie Xia;Ruiqin Xiong;Jing Zhao;Lizhi Wang;Shuyuan Zhu;Xiaopeng Fan;Tiejun Huang","doi":"10.1109/TCI.2025.3561668","DOIUrl":"https://doi.org/10.1109/TCI.2025.3561668","url":null,"abstract":"The acquisition of high-resolution image sequence for dynamic scenes of fast motion remains challenging due to motion blur caused by fast object movement. As a novel neuromorphic sensor, spike camera records the changing light intensity via spike stream of ultra-high temporal resolution, excelling in motion recording but limited in spatial resolution. This paper proposes a method for high spatio-temporal resolution (HSTR) imaging with a hybrid Spike-RGB camera, utilizing the information from spike stream to enhance the temporal resolution and the information from RGB images to enhance the spatial resolution of texture details. For this purpose, we present HSTR-Net, a dedicated network to process the spike and RGB data, which incorporates three key innovations: 1) A temporal control encoder enabling flexible temporal reconstruction through spike stream processing with embedded time parameters, eliminating the requirement to train multiple inference models; 2) Motion-aware feature projection that aligns RGB frame details to target timestamps using spike-derived motion offsets; 3) An adaptive transformer-based fusion strategy establishing cross-modal spatial correlations through mutual attention mechanisms. Extensive experiments demonstrate state-of-the-art performance on synthetic benchmark datasets with 5.23 dB PSNR and 6.94% SSIM improvement. It also shows visually impressive performance on real-world captured spike dataset.","PeriodicalId":56022,"journal":{"name":"IEEE Transactions on Computational Imaging","volume":"11 ","pages":"586-598"},"PeriodicalIF":4.2,"publicationDate":"2025-03-07","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143918692","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Tony G. Allen;David J. Rabb;Gregery T. Buzzard;Charles A. Bouman
{"title":"CLAMP: Majorized Plug-and-Play for Coherent 3D Lidar Imaging","authors":"Tony G. Allen;David J. Rabb;Gregery T. Buzzard;Charles A. Bouman","doi":"10.1109/TCI.2025.3548468","DOIUrl":"https://doi.org/10.1109/TCI.2025.3548468","url":null,"abstract":"Coherent lidar uses a chirped laser pulse for 3D imaging of distant targets. However, existing coherent lidar image reconstruction methods do not account for the system's aperture, resulting in sub-optimal resolution. Moreover, these methods use majorization-minimization for computational efficiency, but do so without a theoretical treatment of convergence. In this paper, we present Coherent Lidar Aperture Modeled Plug-and-Play (CLAMP) for multi-look coherent lidar image reconstruction. CLAMP uses multi-agent consensus equilibrium (a form of PnP) to combine a neural network denoiser with an accurate surrogate forward model of coherent lidar. Additionally, CLAMP introduces a computationally efficient FFT-based method to account for the system's aperture to improve resolution of reconstructed images. Furthermore, we formalize the use of majorization-minimization in consensus optimization problems and prove convergence to the exact consensus equilibrium solution. Finally, we apply CLAMP to synthetic and measured data to demonstrate its effectiveness in producing high-resolution, speckle-free, 3D imagery.","PeriodicalId":56022,"journal":{"name":"IEEE Transactions on Computational Imaging","volume":"11 ","pages":"506-519"},"PeriodicalIF":4.2,"publicationDate":"2025-03-05","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143845339","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
K. Aditya Mohan;Massimiliano Ferrucci;Chuck Divin;Garrett A. Stevenson;Hyojin Kim
{"title":"Distributed Stochastic Optimization of a Neural Representation Network for Time-Space Tomography Reconstruction","authors":"K. Aditya Mohan;Massimiliano Ferrucci;Chuck Divin;Garrett A. Stevenson;Hyojin Kim","doi":"10.1109/TCI.2025.3547265","DOIUrl":"https://doi.org/10.1109/TCI.2025.3547265","url":null,"abstract":"4D time-space reconstruction of dynamic events or deforming objects using X-ray computed tomography (CT) is an important inverse problem in non-destructive evaluation. Conventional back-projection based reconstruction methods assume that the object remains static for the duration of several tens or hundreds of X-ray projection measurement images (reconstruction of consecutive limited-angle CT scans). However, this is an unrealistic assumption for many in-situ experiments that causes spurious artifacts and inaccurate morphological reconstructions of the object. To solve this problem, we propose to perform a 4D time-space reconstruction using a distributed implicit neural representation (DINR) network that is trained using a novel distributed stochastic training algorithm. Our DINR network learns to reconstruct the object at its output by iterative optimization of its network parameters such that the measured projection images best match the output of the CT forward measurement model. We use a forward measurement model that is a function of the DINR outputs at a sparsely sampled set of continuous valued 4D object coordinates. Unlike previous neural representation architectures that forward and back propagate through dense voxel grids that sample the object's entire time-space coordinates, we only propagate through the DINR at a small subset of object coordinates in each iteration resulting in an order-of-magnitude reduction in memory and compute for training. DINR leverages distributed computation across several compute nodes and GPUs to produce high-fidelity 4D time-space reconstructions. We use both simulated parallel-beam and experimental cone-beam X-ray CT datasets to demonstrate the superior performance of our approach.","PeriodicalId":56022,"journal":{"name":"IEEE Transactions on Computational Imaging","volume":"11 ","pages":"362-376"},"PeriodicalIF":4.2,"publicationDate":"2025-03-03","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143716387","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Transfer Learning for Data Fusion for Electromagnetic and Ultrasound Breast Imaging","authors":"Valentin Noël;Thomas Rodet;Dominique Lesselier","doi":"10.1109/TCI.2025.3541934","DOIUrl":"https://doi.org/10.1109/TCI.2025.3541934","url":null,"abstract":"Aiming at improved breast imaging, this contribution explores several scenarios for segmenting and estimating the distribution of electromagnetic (EM) and/or ultrasonic (US) parameters within breast tissue. A two-fold approach is adopted, leveraging Transfer Learning (TL) through Bayesian Neural Networks (BNN); the first objective is to consistently enhance imaging results, and the second is to establish a novel framework for data fusion transfer learning. The methodological approach is tailored for Artificial, Convolutional, and Bayesian Neural Networks, showcasing its effectiveness through the analysis of electromagnetic (EM) and ultrasonic (US) datasets computed in reliable scenarios, with a focus on heterogeneously dense and extremely dense breasts. Furthermore, a novel transfer learning Bayesian data fusion framework incorporating multi-frequency data exploits the complementary nature of EM low-resolution and US high-resolution imaging. By enhancing the fusion of EM and US data, this framework leads to better-contrasted zones in the images and is shown to outperform the most common transfer learning approaches.","PeriodicalId":56022,"journal":{"name":"IEEE Transactions on Computational Imaging","volume":"11 ","pages":"546-555"},"PeriodicalIF":4.2,"publicationDate":"2025-02-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143875175","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Towards Robust and Generalizable Lensless Imaging With Modular Learned Reconstruction","authors":"Eric Bezzam;Yohann Perron;Martin Vetterli","doi":"10.1109/TCI.2025.3539448","DOIUrl":"https://doi.org/10.1109/TCI.2025.3539448","url":null,"abstract":"Lensless cameras disregard the conventional design that imaging should mimic the human eye. This is done by replacing the lens with a thin mask, and moving image formation to the digital post-processing. State-of-the-art lensless imaging techniques use learned approaches that combine physical modeling and neural networks. However, these approaches make simplifying modeling assumptions for ease of calibration and computation. Moreover, the generalizability of learned approaches to lensless measurements of new masks has not been studied. To this end, we utilize a modular learned reconstruction in which a key component is a pre-processor prior to image recovery. We theoretically demonstrate the pre-processor's necessity for standard image recovery techniques (Wiener filtering and iterative algorithms), and through extensive experiments show its effectiveness for multiple lensless imaging approaches and across datasets of different mask types (amplitude and phase). We also perform the first generalization benchmark across mask types to evaluate how well reconstructions trained with one system generalize to others. Our modular reconstruction enables us to use pre-trained components and transfer learning on new systems to cut down weeks of tedious measurements and training. As part of our work, we open-source four datasets, and software for measuring datasets and for training our modular reconstruction.","PeriodicalId":56022,"journal":{"name":"IEEE Transactions on Computational Imaging","volume":"11 ","pages":"213-227"},"PeriodicalIF":4.2,"publicationDate":"2025-02-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143521502","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"Towards Single-Lens Controllable Depth-of-Field Imaging via Depth-Aware Point Spread Functions","authors":"Xiaolong Qian;Qi Jiang;Yao Gao;Shaohua Gao;Zhonghua Yi;Lei Sun;Kai Wei;Haifeng Li;Kailun Yang;Kaiwei Wang;Jian Bai","doi":"10.1109/TCI.2025.3544019","DOIUrl":"https://doi.org/10.1109/TCI.2025.3544019","url":null,"abstract":"Controllable Depth-of-Field (DoF) imaging commonly produces amazing visual effects based on heavy and expensive high-end lenses. However, confronted with the increasing demand for mobile scenarios, it is desirable to achieve a lightweight solution with Minimalist Optical Systems (MOS). This work centers around two major limitations of MOS, i.e., the severe optical aberrations and uncontrollable DoF, for achieving single-lens controllable DoF imaging via computational methods. A Depth-aware Controllable DoF Imaging (DCDI) framework is proposed equipped with All-in-Focus (AiF) aberration correction and monocular depth estimation, where the recovered image and corresponding depth map are utilized to produce imaging results under diverse DoFs of any high-end lens via patch-wise convolution. To address the depth-varying optical degradation, we introduce a Depth-aware Degradation-adaptive Training (DA<inline-formula> <tex-math>$^{2}$</tex-math></inline-formula>T) scheme. At the dataset level, a Depth-aware Aberration MOS (DAMOS) dataset is established based on the simulation of Point Spread Functions (PSFs) under different object distances. Additionally, we design two plug-and-play depth-aware mechanisms to embed depth information into the aberration image recovery for better tackling depth-aware degradation. Furthermore, we propose a storage-efficient Omni-Lens-Field model to represent the 4D PSF library of various lenses. With the predicted depth map, recovered image, and depth-aware PSF map inferred by Omni-Lens-Field, single-lens controllable DoF imaging is achieved. To the best of our knowledge, we are the first to explore the single-lens controllable DoF imaging solution. Comprehensive experimental results demonstrate that the proposed framework enhances the recovery performance, and attains impressive single-lens controllable DoF imaging results, providing a seminal baseline for this field.","PeriodicalId":56022,"journal":{"name":"IEEE Transactions on Computational Imaging","volume":"11 ","pages":"305-320"},"PeriodicalIF":4.2,"publicationDate":"2025-02-28","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143611822","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
{"title":"NLCMR: Indoor Depth Recovery Model With Non-Local Cross-Modality Prior","authors":"Junkang Zhang;Zhengkai Qi;Faming Fang;Tingting Wang;Guixu Zhang","doi":"10.1109/TCI.2025.3545358","DOIUrl":"https://doi.org/10.1109/TCI.2025.3545358","url":null,"abstract":"Recovering a dense depth image from sparse inputs is inherently challenging. Image-guided depth completion has become a prevalent technique, leveraging sparse depth data alongside RGB images to produce detailed depth maps. Although deep learning-based methods have achieved notable success, many state-of-the-art networks operate as black boxes, lacking transparent mechanisms for depth recovery. To address this, we introduce a novel model-guided depth recovery method. Our approach is built on a maximum a posterior (MAP) framework and features an optimization model that incorporates a non-local cross-modality regularizer and a deep image prior. The cross-modality regularizer capitalizes on the inherent correlations between depth and RGB images, enhancing the extraction of shared information. Additionally, the deep image prior captures local characteristics between the depth and RGB domains effectively. To counter the challenge of high heterogeneity leading to degenerate operators, we have integrated an implicit data consistency term into our model. Our model is then realized as a network using the half-quadratic splitting algorithm. Extensive evaluations on the NYU-Depth V2 and SUN RGB-D datasets demonstrate that our method performs competitively with current deep learning techniques.","PeriodicalId":56022,"journal":{"name":"IEEE Transactions on Computational Imaging","volume":"11 ","pages":"265-276"},"PeriodicalIF":4.2,"publicationDate":"2025-02-27","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143570693","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Yao Gao;Qi Jiang;Shaohua Gao;Lei Sun;Kailun Yang;Kaiwei Wang
{"title":"Exploring Quasi-Global Solutions to Compound Lens Based Computational Imaging Systems","authors":"Yao Gao;Qi Jiang;Shaohua Gao;Lei Sun;Kailun Yang;Kaiwei Wang","doi":"10.1109/TCI.2025.3545357","DOIUrl":"https://doi.org/10.1109/TCI.2025.3545357","url":null,"abstract":"Recently, joint design approaches that simultaneously optimize optical systems and downstream algorithms through data-driven learning have demonstrated superior performance over traditional separate design approaches. However, current joint design approaches heavily rely on the manual identification of initial lenses, posing challenges and limitations, particularly for compound lens systems with multiple potential starting points. In this work, we present Quasi-Global Search Optics (QGSO) to automatically design compound lens based computational imaging systems through two parts: (i) Fused Optimization Method for Automatic Optical Design (OptiFusion), which searches for diverse initial optical systems under certain design specifications; and (ii) Efficient Physic-aware Joint Optimization (EPJO), which conducts parallel joint optimization of initial optical systems and image reconstruction networks with the consideration of physical constraints, culminating in the selection of the optimal solution in all search results. Extensive experimental results illustrate that QGSO serves as a transformative end-to-end lens design paradigm for superior global search ability, which automatically provides compound lens based computational imaging systems with higher imaging quality compared to existing paradigms.","PeriodicalId":56022,"journal":{"name":"IEEE Transactions on Computational Imaging","volume":"11 ","pages":"333-348"},"PeriodicalIF":4.2,"publicationDate":"2025-02-25","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143654979","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Ruiming Yu;Hongshan Yu;Haiqiang Xu;Wei Sun;Naveed Akhtar;Yaonan Wang
{"title":"SLBL-PU: Shadow-Based Layer-By-Layer Phase Unwrapping for Efficient 3D Measurement","authors":"Ruiming Yu;Hongshan Yu;Haiqiang Xu;Wei Sun;Naveed Akhtar;Yaonan Wang","doi":"10.1109/TCI.2025.3544084","DOIUrl":"https://doi.org/10.1109/TCI.2025.3544084","url":null,"abstract":"Phase-shifting (PS) based structured light technology shows excellent 3D perception performance. However, it requires projecting a extensive array of patterns, imposing constraints on the measurement space, or embedding additional signals for phase unwrapping (PU), leading to motion artifacts and low robustness. To surmount these challenges, we propose a shadow-based, layer-by-layer phase unwrapping (SLBL-PU) method, which enables absolute phase recovery for deep objects without the need for any supplementary patterns. In the initial phase, attention is focused on a novel truncation feature within the local phase, facilitating the use of iterative PUs to derive the modulated phase. Inspired by shading theory, in the second phase, the absolute phase is restored based on the geometric relationship between the imaging system and the object shadows. Additionally, by incorporating a time-division multiplexing strategy, the efficiency of 3D reconstruction in dynamic scenes is further tripled. In experiments involving different depths, phase modulation, complex colored, and dynamic scenes, the proposed method demonstrated superior performance. Specifically, in static environments (0 mm/s), the proposed approach yields greater measurement accuracy (0.020 mm and 0.195 mm) than does the traditional spatial domain modulation (PS) method. In dynamic environments (15 mm/s), the proposed approach theoretically utilizes at least three patterns, with a defect rate lower than that of the nine-pattern, three-frequency PS method (8.58% and 14.68%).","PeriodicalId":56022,"journal":{"name":"IEEE Transactions on Computational Imaging","volume":"11 ","pages":"452-467"},"PeriodicalIF":4.2,"publicationDate":"2025-02-24","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143792842","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}
Jiawei Dong;Hong Zeng;Sen Dong;Weining Chen;Qianxi Li;Jianzhong Cao;Qiurong Yan;Hao Wang
{"title":"Enhanced Single Pixel Imaging by Using Adaptive Jointly Optimized Conditional Diffusion","authors":"Jiawei Dong;Hong Zeng;Sen Dong;Weining Chen;Qianxi Li;Jianzhong Cao;Qiurong Yan;Hao Wang","doi":"10.1109/TCI.2025.3544087","DOIUrl":"https://doi.org/10.1109/TCI.2025.3544087","url":null,"abstract":"Single-pixel imaging can reconstruct the original image at a low measurement rate (MR), and the target can be measured and reconstructed in low-light environments by capturing the light intensity information using a single-photon detector. Optimizing reconstruction results at low MR has become a focal point of research aimed at enhancing measurement efficiency. The application of neural network has significantly improved reconstruction quality, but the performance still requires further enhancement. In this paper, a Diffusion Single Pixel Imaging Model (DSPIM) method is proposed. The conditional diffusion model is utilized in the training and reconstruction processes of single-pixel imaging and is jointly optimized with an autoencoder network. This approach simulates the measurement and preliminary reconstruction of images, which are incorporated into the diffusion process as conditions. The noises and features are learned through a designed loss function that consists of predicted noise loss and measurement accuracy loss, allowing the reconstruction to perform well at very low MR. Besides, an adaptive regularization coefficients adjustment method (ARCA) has been designed for more effective optimization. Finally, the learned weights are loaded into the single photon counting system as a measurement matrix, demonstrating that the blurriness caused by insufficient features at low MR is effectively addressed using our methods, resulting in clearer targets and well-distinguished features.","PeriodicalId":56022,"journal":{"name":"IEEE Transactions on Computational Imaging","volume":"11 ","pages":"289-304"},"PeriodicalIF":4.2,"publicationDate":"2025-02-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":null,"resultStr":null,"platform":"Semanticscholar","paperid":"143601908","PeriodicalName":null,"FirstCategoryId":null,"ListUrlMain":null,"RegionNum":2,"RegionCategory":"计算机科学","ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":"","EPubDate":null,"PubModel":null,"JCR":null,"JCRName":null,"Score":null,"Total":0}