Double cross-validation for performance evaluation of multi-objective genetic fuzzy systems

Abstract

We propose an idea of using repeated double cross-validation to evaluate the generalization ability of multi-objective genetic fuzzy systems (MoGFS). The main advantage of MoGFS approaches is that a large number of non-dominated fuzzy rule-based systems are obtained by their single run. Each of the obtained fuzzy rule-based systems has a different tradeoff with respect to conflicting objectives such as accuracy and complexity. One controversial issue in the MoGFS field (and also in the field of multi-objective optimization in general) is how to choose the final solution from the obtained non-dominated ones. Since this selection is supposed to be done by human users, it is very difficult to rigorously discuss the generalization ability of the finally-selected fuzzy rule-based system. To tackle this difficulty, we propose the use of double cross-validation in the performance evaluation of MoGFS approaches. Double cross-validation has a nested structure of two cross-validation loops. The inner loop is used to determine the best complexity of fuzzy rule-based systems with the highest generalization ability for the training data in each run in the outer loop. That is, the inner loop plays the role of validation data. The determined best complexity is used to choose the final fuzzy rule-based system in each run in the outer loop. We explain the proposed idea by applying it to the performance evaluation of fuzzy rule-based classifiers designed by our multi-objective fuzzy genetics-based machine learning algorithm.