Event-Enhanced Snapshot Mosaic Hyperspectral Frame Deblurring

Mengyue Geng;Lizhi Wang;Lin Zhu;Wei Zhang;Ruiqin Xiong;Yonghong Tian
{"title":"Event-Enhanced Snapshot Mosaic Hyperspectral Frame Deblurring","authors":"Mengyue Geng;Lizhi Wang;Lin Zhu;Wei Zhang;Ruiqin Xiong;Yonghong Tian","doi":"10.1109/TPAMI.2024.3465455","DOIUrl":null,"url":null,"abstract":"Snapshot Mosaic Hyperspectral Cameras (SMHCs) are popular hyperspectral imaging devices for acquiring both color and motion details of scenes. However, the narrow-band spectral filters in SMHCs may negatively impact their motion perception ability, resulting in blurry SMHC frames. In this paper, we propose a hardware-software collaborative approach to address the blurring issue of SMHCs. Our approach involves integrating SMHCs with neuromorphic event cameras for efficient event-enhanced SMHC frame deblurring. To achieve spectral information recovery guided by event signals, we formulate a spectral-aware Event-based Double Integral (sEDI) model that links SMHC frames and events from a spectral perspective, providing principled model design insights. Then, we develop a Diffusion-guided Noise Awareness (DNA) training framework that utilizes diffusion models to learn noise-aware features and promote model robustness towards camera noise. Furthermore, we design an Event-enhanced Hyperspectral frame Deblurring Network (EvHDNet) based on sEDI, which is trained with DNA and features improved spatial-spectral learning and modality interaction for reliable SMHC frame deblurring. Experiments on both synthetic data and real data show that the proposed DNA + EvHDNet outperforms state-of-the-art methods on both spatial and spectral fidelity. The code and dataset will be made publicly available.","PeriodicalId":94034,"journal":{"name":"IEEE transactions on pattern analysis and machine intelligence","volume":"47 1","pages":"206-223"},"PeriodicalIF":0.0000,"publicationDate":"2024-09-20","publicationTypes":"Journal Article","fieldsOfStudy":null,"isOpenAccess":false,"openAccessPdf":"","citationCount":"0","resultStr":null,"platform":"Semanticscholar","paperid":null,"PeriodicalName":"IEEE transactions on pattern analysis and machine intelligence","FirstCategoryId":"1085","ListUrlMain":"https://ieeexplore.ieee.org/document/10684998/","RegionNum":0,"RegionCategory":null,"ArticlePicture":[],"TitleCN":null,"AbstractTextCN":null,"PMCID":null,"EPubDate":"","PubModel":"","JCR":"","JCRName":"","Score":null,"Total":0}
引用次数: 0

Abstract

Snapshot Mosaic Hyperspectral Cameras (SMHCs) are popular hyperspectral imaging devices for acquiring both color and motion details of scenes. However, the narrow-band spectral filters in SMHCs may negatively impact their motion perception ability, resulting in blurry SMHC frames. In this paper, we propose a hardware-software collaborative approach to address the blurring issue of SMHCs. Our approach involves integrating SMHCs with neuromorphic event cameras for efficient event-enhanced SMHC frame deblurring. To achieve spectral information recovery guided by event signals, we formulate a spectral-aware Event-based Double Integral (sEDI) model that links SMHC frames and events from a spectral perspective, providing principled model design insights. Then, we develop a Diffusion-guided Noise Awareness (DNA) training framework that utilizes diffusion models to learn noise-aware features and promote model robustness towards camera noise. Furthermore, we design an Event-enhanced Hyperspectral frame Deblurring Network (EvHDNet) based on sEDI, which is trained with DNA and features improved spatial-spectral learning and modality interaction for reliable SMHC frame deblurring. Experiments on both synthetic data and real data show that the proposed DNA + EvHDNet outperforms state-of-the-art methods on both spatial and spectral fidelity. The code and dataset will be made publicly available.
事件增强快照马赛克高光谱帧去模糊。
快照马赛克高光谱相机(SMHC)是一种流行的高光谱成像设备,用于获取场景的色彩和运动细节。然而,SMHC 中的窄带光谱滤波器可能会对其运动感知能力产生负面影响,从而导致 SMHC 图像模糊。在本文中,我们提出了一种软硬件协同方法来解决 SMHC 的模糊问题。我们的方法涉及将 SMHC 与神经形态事件相机集成,以实现高效的事件增强 SMHC 帧去模糊。为了在事件信号的引导下实现光谱信息恢复,我们提出了光谱感知的基于事件的双积分(sEDI)模型,该模型从光谱角度将 SMHC 帧和事件联系起来,提供了原则性的模型设计见解。然后,我们开发了扩散引导噪声感知(DNA)训练框架,利用扩散模型学习噪声感知特征,提高模型对摄像机噪声的鲁棒性。此外,我们还设计了基于 sEDI 的事件增强型高光谱帧去模糊网络(EvHDNet),该网络使用 DNA 进行训练,具有改进的空间-光谱学习和模态交互功能,可用于可靠的 SMHC 帧去模糊。在合成数据和真实数据上的实验表明,拟议的 DNA + EvHDNet 在空间和光谱保真度上都优于最先进的方法。代码和数据集将公开发布。
本文章由计算机程序翻译,如有差异,请以英文原文为准。
求助全文
约1分钟内获得全文 求助全文
来源期刊
自引率
0.00%
发文量
0
×
引用
GB/T 7714-2015
复制
MLA
复制
APA
复制
导出至
BibTeX EndNote RefMan NoteFirst NoteExpress
×
提示
您的信息不完整,为了账户安全,请先补充。
现在去补充
×
提示
您因"违规操作"
具体请查看互助需知
我知道了
×
提示
确定
请完成安全验证×
copy
已复制链接
快去分享给好友吧!
我知道了
右上角分享
点击右上角分享
0
联系我们:info@booksci.cn Book学术提供免费学术资源搜索服务,方便国内外学者检索中英文文献。致力于提供最便捷和优质的服务体验。 Copyright © 2023 布克学术 All rights reserved.
京ICP备2023020795号-1
ghs 京公网安备 11010802042870号
Book学术文献互助
Book学术文献互助群
群 号:481959085
Book学术官方微信