Fuzzy Inference Systems Applied to the Combining Regression Models

Ensemble regression systems combine regression models expecting that several models outperform any single base model in prediction accuracy. To achieve a common prediction, individual models are combined using two techniques: model selection and ensemble integration. The concept of model accuracy in the input space plays the crucial role in the dynamic selection and integration of base models. The model accuracy for any test sample in the input space can be calculated in a two-step procedure which is a typical scheme of supervised machine learning. First, the accuracy set, i.e., the set of model accuracies for all validation samples is created. Then, on the base of the accuracy set, accuracy measure (function) is constructed using a supervised learning method. In this paper, the second step of the procedure is addressed by constructing accuracy measure of base model using fuzzy inference engines. Two methods are developed and applied in the ensemble system with dynamic scheme of models selection and integration: Mamdani and Takagi-Sugeno-Kang fuzzy inference systems. Both fuzzy inference systems were experimentally tested and compared against 6 literature methods of combining base models using 25 benchmark databases and three homogeneous pools of base models, containing multilayer perceptrons, 5-nearest neighbor models and linear regression models. The experimental results clearly show the effectiveness of the proposed supervised learning algorithm using fuzzy reasoning systems for dynamic selection and integration of the homogeneous ensemble of base models.