Chestnut (Castanea sativa) is a nutritious food with fiber, vitamins C and B group, minerals such as potassium, magnesium, and iron. In addition to being a nutritious food, chestnuts are used in various fields such as medicine, cosmetics, and energy. All the mentioned characteristics make it a demanded product worldwide. To determine the market price of chestnuts, it is necessary to have a good classification. In traditional approaches, producers classify chestnuts according to their external appearance; however, this is tedious, time-consuming, and prone to errors. There is a need for computer-aided systems to analyze the chestnut varieties. Therefore, a camera system was set up and images of chestnuts belonging to ‘Alandız’, ‘Aydın’, ‘Simav’, and ‘Zonguldak’ varieties were captured to create a novel dataset. Moreover, a deep-based mobile application was developed to classify chestnut types. After testing the 16 state-of-the-art convolutional neural network (CNN) models, the three most successful models from the 16 CNN models were used as feature extractors, and the extracted features were classified using Decision Tree (DT), Naive Bayes (NB), Support Vector Machine (SVM), Adaboost, and Xtreme Gradient Boosting (XGB) algorithms. Finally, attention modules were integrated to CNN models to enhance accurate classification of chestnut images. The highest result achieved by MobileNet with attention mechanism was accuracy of 99.65%, precision of 99.62%, recall of 99.67%, f1-score of 99.64%, kappa score of 100%, and area under the curve (AUC) of 100%. The chestnut dataset can be used in literature studies for different purposes and the proposed framework can be utilized as a computer-aided decision support system for experts in farming.