Clustering of unlabeled data is a critical task for extracting meaningful patterns from large and complex datasets. In this context, metaheuristic optimization-based clustering methods have gained popularity due to their ability to handle nonlinear and high-dimensional search spaces. This study introduces the Hybrid Fox Optimization Algorithm (ECFOX), an improved optimization and clustering method that builds upon the standard FOX algorithm. ECFOX integrates chaotic maps for population initialization and adaptive control, as well as a polynomial mutation operator to enhance solution diversity and local refinement. The Singer chaotic map generates a well-distributed initial population, while the Iterative chaotic map adaptively balances exploration and exploitation. A polynomial mutation operator is periodically applied to refine candidate solutions and maintain diversity. The effectiveness of ECFOX was evaluated in two experimental stages. First, clustering performance was tested on 17 real-world datasets from the UCI Machine Learning Repository, comparing ECFOX with traditional clustering methods (K-means, K-medoids, Fuzzy C-means) and popular metaheuristic algorithms (ChOA, GWO, WOA, PSO, FOX). ECFOX achieved superior results on most datasets. In the second stage, ECFOX was tested on 23 classical benchmark functions to assess its global and local search performance. The results were compared with those of well-known metaheuristic algorithms, including GWO, CHIMP, PSO, ALO, IALO, and the standard FOX algorithm. ECFOX demonstrated superior convergence speed, solution quality, and robustness. Statistical validation using Wilcoxon signed-rank and Friedman tests confirmed the significance of ECFOX’s improvements. These findings suggest that ECFOX is a reliable and competitive approach for clustering and general optimization problems.