Improvement of Overfitting Problem in Ensemble Classifiers

Abstract

A challenging topic in designing and training ensemble classifiers is overfitting. Evidently, employing multiple complex classifiers may increase the success of ensemble classifiers in feature space division with intertwined data and also decrease the training error to a minimum value. However, this success does not exist in test data. Ensemble classifiers are more potential to involve overfitting than single classifiers; because, overfitting occurrence in each base classifier can spread this problem to the final decision of the ensemble. In this paper, an approach is proposed based on heuristic methods in order to improve overfitting. In this way, Multi-Objective Inclined Planes Optimization (MOIPO) algorithm, Multi-Objective Particle Swarm Optimization (MOPSO) algorithm, and Multi-Objective Gravitational Search Algorithm (MOGSA) are applied, and their results are compared with each other. Experimental results on various types of datasets with nonlinear, overlapping class boundaries and different feature space dimensions confirm that the concurrent minimization of ensemble size and error rate in the training phase can lead to a significant reduction in the amount of overfitting. In fact, with this approach during the training phase, the ensemble classifier is obliged to minimize the error with the least and most simple base classifiers, and therefore overfitting is inevitably avoided. The obtained results on test data certify the supremacy of MOIPO as a new method. Finally, the comparison of error rate of proposed method with other methods related to overfitting improvement shows that MOIPO based method performs better and leads to an improvement of error rate by 76.7%.