UnSP: Improving event-to-image reconstruction with uncertainty guided self-paced learning

Asynchronous events, produced by event cameras, possess several advantages against traditional cameras: high temporal resolution, dynamic range, etc. Traditional event-to-image reconstruction methods adopt computer vision techniques and establish a correspondence between event streams and the reconstruction image. Despite great successes, those methods ignore filtering the non-confident event frames, and hence produce unsatisfactory reconstruction results. In this paper, we propose a plug-and-play model by using uncertainty guided self-paced learning (dubbed UnSP) for finetuning the event-to-image reconstruction process. The key observation of UnSP is that, different event streams, though corresponding to a common reconstruction image, serve as different functions during the training process of event-to-image reconstruction networks (e.g., shape, intensity, details are extracted in different training phases of networks). Typically, UnSP proposes an uncertainty modeling for each event frame based on its reconstruction errors induced by three metrics, and then filters confident event frames in a self-paced learning fashion. Experiments on the six subsets of the Event Camera Dataset shows that UnSP can be incorporated with any event-to-image reconstruction networks seamlessly and achieve significant improvement in both quantitative and qualitative results. In summary, the uncertainty-driven adaptive sampling and self-learning mechanisms of UnSP, coupled with its plug-and-play capability, enhance the robustness, efficiency, and versatility for event-to-image reconstruction. Code is available at https://github.com/wangsfan/UnSP.