Accurate classification of adrenal lesions on magnetic resonance (MR) images are very important for diagnosis and treatment planning. The detection and classification of lesions in medical imaging heavily rely on several key factors, including the specialist's level of experience, work intensity, and fatigue of the clinician. These factors are critical determinants of the accuracy and effectiveness of the diagnostic process, which in turn has a direct impact on patient health outcomes. With the spread of artificial intelligence, the use of computer-aided diagnosis (CAD) systems in disease diagnosis has also increased. In this study, adrenal lesion classification was performed using deep learning on MR images. The data set used was obtained from the Department of Radiology, Faculty of Medicine, Selcuk University, and all adrenal lesions were identified and reviewed in consensus by two radiologists experienced with abdominal MR. Studies were carried out on two different data sets created by T1- and T2-weighted MR images. The data set consisted of 112 benign and 10 malignant lesions for each mode. Experiments were performed with regions of interest (ROIs) of different sizes to increase the working performance. Thus, the effect of the selected ROI size on the classification performance was assessed. In addition, instead of the convolutional neural network (CNN) models used in deep learning, a unique classification model structure called Abdomen Caps was proposed. When the data sets used in classification studies are manually separated for training, validation, and testing, different results are obtained with different data sets for each stage. To eliminate this imbalance, tenfold cross-validation was used in this study. The best results obtained were 0.982, 0.999, 0.969, 0.983, 0.998, and 0.964 for accuracy, precision, recall, F1-score, area under the curve (AUC) score, and kappa score, respectively.