Rethinking Uncertainty Measures: Replacing Entropy with Extropy in Learning Algorithms

Abstract

This paper investigates replacing entropy with extropy as the impurity measure in decision tree algorithms. Decision trees are widely used and interpretable machine learning models that recursively partition data based on features to create hierarchical decision rules. Impurity measures such as entropy and the Gini index play a key role in selecting the best feature and split point at each node. Extropy, a newer measure focusing on the probability of error rather than uncertainty, has been proposed as an alternative to entropy. Using the Iris dataset, the performance of decision trees constructed with entropy, extropy, and the Gini index was compared. Results showed that the entropy-based tree achieved the highest accuracy of approximately 97.78%, while extropy- and Gini-based trees both attained similar accuracies around 93.33%. The performance difference between entropy and Gini is generally small, and due to its computational efficiency, the Gini index is often preferred in practical applications. Extropy, with its focus on error probability, demonstrated competitive performance close to Gini and can serve as a suitable alternative in scenarios where errorfocused impurity assessment is desirable