On ReLU neural networks as piecewise linear surrogate models

Abstract

Continuous piecewise linear (CPWL) surrogate models are increasingly used in process systems engineering to represent complex, nonlinear relationships. Neural networks with ReLU activation functions (ReLU-NN) have become a common method to represent CPWL models. However, the structure and behavior of the linear partitions formed by rectifier networks have not been fully examined. In this study, we propose exact mathematical expressions for linear functions and linear regions of small rectifier networks. Moreover, we analyze the performance of the rectifier networks from a polyhedral perspective and introduce the three major challenges associated with these models: redundancy, degeneracy, and low efficiency. Furthermore, we assess difference-of-convex continuous piecewise linear (DC-CPWL) function as an alternative representation of CPWL relationships and compare it to ReLU-based shallow and deep Neural Networks across four industrial case studies. Our findings demonstrate that the DC-CPWL representation consistently yields highly efficient models while the ReLU-NN representation generates less efficient ones.