Background: Prostate cancer (PCa) has the second-highest morbidity and mortality rates in men. Possessing facile surface chemistry and unique optical properties make silica nanoparticles(SiO2-NPs) promis-ing cancer therapy materials. Objective: This study aimed to investigate the effects of SiO2-NPs and their derivatives, including SiNP-NH2, SiNP-Cl, and SiNP-SH against PCa and clarify their molecular mechanism on cell death, gene, and protein ex-pressions. Methods: Following the synthesis and derivation of SiO2-NPs, their characterization was carried out using TEM, DLS, BET, and FT-IR. Cytotoxic properties of the compounds were investigated against different human cancerous cells; including HUH-7, A549, DLD-1, HeLa, NCI-H295R, and PC-3, as well as human healthy epithelium cell line PNT1A. Results: SiNP-NH2, SiNP-Cl, and SiNP-SH dose-dependently inhibited the proliferation of PC-3 cells with an IC50 value as 55.46 μg/mL, 55.09 μg/mL and 72.89 μg/mL, respectively. SiNP-SH significantly(p<0.0001) inhibited metastasis and invasion of PC-3 cells(20.4% and 46.7%, respectively), and significantly(p<0.0001) increased early apoptosis(32.3%) when compared with non-treated cells. Protein and mRNA expressions of BcL-2, Bax, caspase-3, caspase-9, caspase-12, p53, Smad-4, Kras, and Nf-ĸB were also altered following the treatment of SiO2-NPs and its derivatives. Conclusion: Our results demonstrated that-SH functioned SiO2-NPs can prevent the proliferation of human PCa by increasing apoptosis by up-regulating gene and protein expression of p53(TP53) as well as caspase-3, caspase-9, and caspase-12 in the apoptotic pathway. Besides, the increased level of Smad-4 has also implicated the decreased cell proliferation. Hence, low sized SiNP-SH nanoparticles might be a suitable candidate for the treatment of human PCa.