This study presents a new evaluation framework that systematically considers the variability in real-life fruit detection, which is of critical importance for agricultural automation. The dataset used comprises 5025 images of oranges captured under nine different environmental conditions, including different phases of the day (morning, afternoon, evening, night) and various weather conditions (sunny, cloudy, rainy, and indoors). The data are split into 70% training (3513), 20% validation (1001), and 10% testing (511) sets to preserve the environmental differences. We compare two members of the latest You Only Look Once (YOLO) family, YOLOv11: the lightweight YOLOv11n for resource-constrained real-time applications and the large-capacity YOLOv11x for higher accuracy. Ultralytics-based training (PyTorch) uses geometric and photometric augmentations, as well as composite augmentations such as mosaic and mixup. Experiments with early stopping were conducted for 50 epochs with input resolutions of 640 × 640 (YOLOv11n) and 896 × 896 (YOLOv11x). On the test set, YOLOv11n achieved P = 0.85, R = 0.74, mAP@50 = 0.83, and mAP@50–95 = 0.46, while YOLOv11x achieved P = 0.86, R = 0.77, mAP@50 = 0.85, and mAP@50–95 = 0.49. In ablation experiments where the augmentations were removed, recall and mAP@50–95 decreased significantly; increasing the input resolution from 640 to 896 improved the localization quality, especially for small/partially occluded objects. Data imbalance (a lack of rainy and nighttime subsets) was found to limit performance. While mosaic/mixup mitigates this effect to some extent, it is argued that targeted oversampling and synthetic data generation may be necessary. On the explainability front, heatmaps obtained with Eigen-CAM show that YOLOv11x produces more consistent spatial attention in low-light and reflectance conditions, while YOLOv11n focuses more on high-contrast edges. The results indicate that YOLOv11n offers a sufficient accuracy/speed balance for real-time tracking on edge devices, while YOLOv11x provides more reliable detection by reducing misses in challenging conditions. Future work is recommended to include synthetic data and domain adaptation, environment-aware sampling, multi-class citrus scenarios, and deployment-focused evaluations on embedded hardware.