Hybrid optimized PSO-CatBoost framework for high-accuracy cell-type classification and identification in single-cell RNA-Seq data

Abstract

Single-cell RNA sequencing (scRNA-seq) has revolutionized our ability to characterize cellular diversity at an unprec edented resolution. However, the classification of cell types from scRNA-seq data remains a challenging task due to the high dimensionality, sparsity, and noise inherent in these datasets. Traditional machine learning approaches often strug gle to achieve high performance and generalizability without extensive manual hyperparameter tuning. A novel hybrid framework that combines Particle Swarm Optimization (PSO) with Categorical Boosting (CatBoost), a gradient boosting decision tree algorithm, is introduced for automated and optimized cell-type classification. PSO is employed to search the hyperparameter space of CatBoost efficiently, enabling the framework to adapt to the unique structure of the scRNA seq data. The proposed PSO-CatBoost framework is evaluated using annotated benchmark datasets, and performance is assessed through multiple evaluation metrics, including accuracy, F1-score, and Area Under the Curve, using k-fold cross-validation. Superior performance was achieved by the PSO-CatBoost framework compared to standard CatBoost models optimized via grid search and random search. Strong results were demonstrated by the proposed method in terms of recall, maintaining high sensitivity across diverse and imbalanced cell-type classes. Visualization of classification out comes and feature importance highlighted the framework’s capacity to focus on biologically relevant gene signatures for each cell type. It is demonstrated that the integration of PSO with CatBoost yields a highly accurate and scalable clas sifier for cell-type prediction in scRNA-seq data. This approach reduces the need for manual hyperparameter tuning and improves performance across a range of cell types, suggesting valuable applications in computational biology, biomedical diagnostics, and large-scale cell atlas projects.