Deep-layer limit and stability analysis of the basic forward–backward-splitting induced network (I): feed-forward systems

Abstract

Forward-backward splitting (FBS) is one of the most fundamental and efficient optimization algorithms in linear inverse problems like sparse recovery and image reconstruction, and has recently been unrolled to construct several deep neural networks with dramatic performance advantages over conventional methods. This circumstance motivates us to consider some theoretical aspects of the basic FBS-induced network. Here, ‘basic’ means that the neural network is unrolled from the original FBS algorithm with direct parameter relaxation. In this paper we report the first part of our study, i.e., deep-layer limit behavior and stability of feed-forward systems. We formulate the finite layer network as a difference inclusion and model the related deep-layer limit system as a differential inclusion. We mainly analyze the uniform convergence properties from the state of the finite layer network to that of the related deep-layer limit system, as well as their forward stability. Our analysis procedure can be simplified to analyze the LISTA- and ALISTA-like networks. A numerical example is implemented to illustrate the convergence results and perturbation stability. As a side product of this study, some corollaries in the case of pointwise sampling and Lipschitz continuity assumptions provide convergence results in the context of numerical ordinary differential inclusion.