Feature selection aims to enhance classification performance by identifying the most relevant attributes in high-dimensional datasets. This study provides a comprehensive evaluation of ten feature selection methods across 27 data scenarios varying in feature count, class number, sample size, and class imbalance. Metaheuristic algorithms Artificial Bee Colony (ABC), Particle Swarm Optimization (PSO), Genetic Algorithm, Differential Evolution (DE), and Simulated Annealing (SA) are compared with traditional methods such as Support Vector Machines, Forward Feature Selection (FFS), Least Absolute Shrinkage and Selection Operator (LASSO) (L1 Regularization), Recursive Feature Elimination, and Random Forest (RF). In addition to extensive simulation-based experiments, the proposed framework is further validated using real-world benchmark dataset to assess practical applicability. Performance is rigorously evaluated via 5-fold cross-validation using Cohen’s Kappa, Macro F1, Matthews Correlation Coefficient and Balanced Accuracy, metrics particularly suitable for imbalanced classification tasks. The results provide valuable insights into the robustness and effectiveness of different feature selection strategies under varying data complexities, offering practical guidance for improving classification model performance.