Search space boundaries in neural network error landscape analysis

Abstract

Fitness landscape analysis encompasses a selection of techniques designed to estimate the properties of a search landscape associated with an optimisation problem. Applied to neural network training, fitness landscape analysis can be used to establish the link between the shape of the objective function and various neural network design and architecture properties. However, most fitness landscape analysis metrics rely on search space sampling. Since neural network search space is unbounded, it is unclear what subset of the search space should be sampled to obtain representative measurements. This study analyses fitness landscape properties of neural networks under various search space boundaries, and proposes meaningful search space bounds for neural network fitness landscape analysis.