Hazelnut is a shrubby plant well-suited to the temperate and humid climate of the Black Sea region, particularly known for its mild winters. Turkey plays a critical role in global hazelnut production. Hazelnuts are not only an essential agricultural product for the country’s economy but are also highly valued for their nutritional content, being particularly rich in healthy oils and proteins, and their associated health benefits. The classification of different hazelnut species using modern deep learning techniques aimed to automatically distinguish between eight commonly cultivated hazelnut types: caklidak, damat, devedisi, sivri, karafindik, palaz, tombul, and yagli. This study employed both YOLOv8 classification models and Vision Transformer (ViT) with fine-tuning to classify a dataset comprising 2,722 labeled images of these hazelnut types. Various configurations of YOLOv8 models were tested, specifically YOLOv8n-cls, YOLOv8s-cls, YOLOv8m-cls, YOLOv8l-cls, and YOLOv8x-cls, alongside ViT-Base/16 with systematic fine-tuning optimization. The results demonstrate that the fine-tuned Vision Transformer achieved the highest classification accuracy of 99.75%, establishing a new state-of-the-art performance for this dataset. Among YOLOv8 models, YOLOv8s-cls and YOLOv8l-cls achieved 99.25% accuracy, while the lightest model, YOLOv8n-cls, recorded 97.38% accuracy. The superior performance of ViT-Base/16 (99.75%) represents a significant advancement over previous studies, demonstrating the effectiveness of transformer-based architectures with transfer learning for agricultural image classification. These findings highlight the strong potential for reliable use in real-world agricultural applications, with the Vision Transformer providing near-perfect classification accuracy. This study shows that advanced computer vision, particularly transformer-based models with fine-tuning, can effectively automate the identification and quality control of agricultural products, enhancing efficiency in food processing and supply chains.