The cytotoxic properties of cobalt oxide (Co3O4) nanoparticles (NPs), in addition to graphene oxide (GO)-Co3O4 and silver (Ag)-GO-Co3O4 nanocomposites (NCs), were evaluated against both human healthy lung fibroblast (MRC-5) and hepatocellular carcinoma (HepG2) cell lines utilizing the XTT assay. The investigation revealed that synthesized Co3O4 NPs and NCs (GO-Co3O4 and Ag-GO-Co3O4) elicited significant cytotoxic responses in MRC-5 and HepG2 cell lines in a concentration-dependent manner. Through molecular docking analyses, it was observed that all fabricated nanomaterials exhibited DNA recognition via minor groove binding, with molecular affinities ranging from − 4.82 to -11.66 kcal/mol. Furthermore, the docking outcomes illustrated that the angular conformations of GO-Co3O4 and Ag-GO-Co3O4 conferred ‘shape-selective’ characteristics as DNA minor groove binders, leading to heightened cytotoxicity, particularly in the HepG2 cell line compared to the normal MRC-5 cell line.