Based on the sample entropy algorithm in nonlinear dynamics, an improved sample entropy method is proposed in the aerodynamic system instability identification for the stall precursor detection based on the nonlinear feature extraction algorithm in an axial compressor. The sample entropy algorithm is an improved algorithm based on the approximate entropy algorithm, which quantifies the regularity and the predictability of data in time series. Combined with the spatial modes representing for the rotating stall in the circumferential direction, the recognition capacity of the sample entropy is displayed well on the detection of stall inception. The indications of rotating waves are extracted by the circumferential analysis from modal wave energy. The significant ascendant in the amplitude of the spatial mode is a pronounced feature well before the imminence of stall. Data processing with the spatial mode effectively avoids the problems of inaccurate identification of a single measuring point only depending on pressure. Due to the different selections of similarity tolerance, two kinds of sample entropy are obtained. The properties of the development process of the identification model show obvious mutation phenomena at the boundary of instability, which reveal the inherent characteristic in aerodynamic system. Then the dynamic difference quotient is computed according to the difference quotient criterion, after the smooth management by discrete wavelet. The rapid increase of difference quotient can be regarded as a significant feature of the system approaching the flow instability. It is proven that based on the principle of sample entropy algorithm, the nonlinear characteristic of rotating stall can be well described. The inception can be suggested by about 12–68 revolutions before the stall arrival. This prediction method presenting is accounted for the nonlinearity of the complex flow in stall, which is in a view of data fusion system of pressure for the spatial mode tracking.