1. Introduction
Amidst escalating global concerns over food safety [
1], raisins—ubiquitous as a popular snack in everyday life—frequently encounter the challenge of foreign object contamination during their production [
2]. Such contamination not only undermines the product’s market competitiveness but also poses a significant risk to consumer health [
3]. Consequently, the advancement of a precise and efficient detection system for foreign objects in raisins is imperative for safeguarding food safety and bolstering product quality.
In conventional raisin production, the identification of foreign matter predominantly depends on manual visual inspection, a method that, despite its simplicity, is fraught with inefficiencies [
4,
5]. Initially, this approach necessitates substantial labor, escalating production expenses and proving impractical for large-scale operations [
6,
7]. Moreover, the efficacy of manual inspection is constrained by factors such as the inspector’s visual acuity, fatigue, and concentration span, which can result in both oversights and erroneous identifications [
8]. In addition, manual detection is difficult to adapt to rapidly changing production environments, and it is hard for inspectors to promptly find foreign bodies that are small or similar in color to raisins.
In stark contrast, automated foreign object detection systems have significant advantages. Initially, automation significantly enhances detection velocity, facilitating real-time surveillance and aligning with the exigencies of high-throughput production lines [
9]. Moreover, these systems, leveraging machine vision, refine their discernment of foreign object characteristics through extensive training datasets, thereby bolstering detection precision and dependability. Immune to the vicissitudes of human fatigue and concentration, they operate with unwavering consistency. Furthermore, with the capacity to fine-tune algorithmic parameters and training data, these automated systems adeptly accommodate diverse production settings and product varieties, adeptly managing a spectrum of foreign object detection challenges [
10].
Numerous researchers have extensively investigated the automation of raisin grading and sorting. Raihen M.N. et al. [
11] designed a new model for raisin classification, combining eleven machine learning and five AI methods to classify raisins based on size, shape, and texture, with the LightGBM model peaking at 98.40% accuracy. Backes A.R. et al. [
12] optimized feature extraction for raisin evaluation through particle swarm algorithms, coupling with linear discriminant analysis (LDA) to reach the zenith of classification accuracy at 99.73%. Predominantly, these raisin detection studies are rooted in traditional machine learning, heavily dependent on feature engineering, and often constrained by oversimplified image backgrounds, limiting their detection efficacy and hindering industrial scalability.
As deep learning technology rapidly develops, an influx of researchers have delved into the detection of agricultural products utilizing this advanced technology. Gao F. et al. [
13] introduced a deep-learning-based multi-class fruit detection approach for plant analysis, employing Faster R-CNN to discern apples under various occlusion scenarios. The model’s accuracy is 0.909 in the absence of obstructions, 0.899 when leaves are blocking the view, 0.858 when branches or wire mesh are obstructing, and 0.848 when other fruits are in the way. The average overall accuracy is 0.879. Tian Y. et al. [
14] proposed an enhanced YOLO-V3 model for the real-time tracking of apples throughout different growth phases in orchards. By integrating DenseNet to refine the YOLO-V3 network’s lower-resolution feature layers, the YOLOV3-dense model achieved an average precision of 0.882, with a swift average detection time of 0.304 s per frame. Ma N. et al. [
15] have developed a technique for detecting and quantifying wheat seeds using an enhanced YOLOv8 model, incorporating shared convolutional layers for a streamlined design with reduced parameters. The integration of a visual transformer equipped with a deformable attention mechanism into the C2f module aims to bolster the network’s feature extraction prowess and improve detection precision. The model demonstrated an average detection accuracy of 77.6% in scenarios with seed aggregation and occlusion, an inference time of 2.86 ms, and improved seed detection accuracy under conditions of stacking, adhesion, and occlusion. Presently, deep learning-based detection techniques are primarily categorized into two-stage object detection methods, exemplified by Faster R-CNN, and single-stage object detection methods, typified by YOLO [
16]. Two-stage methods boast higher accuracy but compromise on real-time capabilities. Conversely, single-stage methods, albeit with marginally lower accuracy, excel in real-time performance and are extensively applied in industrial settings.
The main goal of this research is to create and deploy a system for detecting and identifying foreign objects in raisins, utilizing an enhanced version of the YOLOv8 algorithm. By deeply analyzing the shortcomings of the YOLOv8 algorithm in handling small target detection and complex background recognition, we propose an enhanced real-time detection approach leveraging an improved YOLOv8 model. Our major contributions are as follows:
Incorporated the multi-head self-attention mechanism (MHSA) from BoTNet into the backbone network of the YOLOv8 model, strengthening the model backbone’s global information capture capability to enhance the richness and completeness of feature extraction.
The Conv module of the neck layer was changed to the RFAConv module, and the feature extraction process of different regions was fine-adjusted, so that the model could dynamically generate acceptance fields and improve the detection ability of foreign bodies of different sizes.
The original network’s CIoU loss is substituted with EIoU, enhancing the efficiency and accuracy of raisin foreign body detection.
Through such algorithm optimization and model adjustments, the detection system’s accuracy and real-time performance have been improved without increasing the number of parameters, making it more suitable for the raisin processing production environment. The subsequent sections of this paper are structured as follows:
Section 2 primarily introduces the dataset construction process used for raisin foreign matter detection, demonstrating the model employed in this paper’s detection method and the significance of each module in the improved model for raisin detection.
Section 3 outlines the experimental setup and parameter settings, detailing the model performance metrics for evaluation. It further presents the training outcomes and detection visualizations and underscores the model’s performance and the efficacy of the refined model through comparative studies with other models, ablation analyses, and experiments on various foreign matter compositions.
Section 4 discusses and analyzes the research in conjunction with other researchers and the research presented in this paper.
Section 5 renders a thorough encapsulation of the entire research undertaking.
3. Results
3.1. Experimental Platform and Parameter Configuration
Table 1 displays the comprehensive setup and specifications of the training platform utilized for model development.
The image size for model training is 640 × 640, the training epochs are 300, the batch size is 32, the learning rate is 0.01, the SGD momentum is 0.937, the weight decay for the optimizer is 0.0005, the warm-up period is 3, the initial momentum during warm-up is 0.8, and the initial learning rate for biases during warm-up is 0.1.
3.2. Performance Evaluation Metrics
For raisin foreign matter detection in scenarios with multiple foreign matters and dense objects, it is necessary to measure the model’s accuracy and speed. Therefore, this paper selects multi-faceted performance evaluation metrics to analyze the model. The primary assessment criteria encompass precision (P), recall (R), F1-score, mean average precision (mAP) across all categories, mean average precision (AP) for individual categories, the count of model parameters, floating-point operations per second (FLOPs), and the duration for image detection on the GPU platform (Time).
The formulas for each performance evaluation metric are shown in Equations (9) to (12), where k represents the number of classes. P indicates the ratio of true positives (TP) to the total predicted positives (TP and FP), while R reflects the ratio of true positives (TP) to the sum of true positives and false negatives (TP and FN). The F1 score is the harmonic average of precision and recall. mAP stands for the mean average precision across multiple classes, and AP denotes the area under the precision–recall curve. The count of model parameters signifies its size, FLOPs measure the computational complexity, and Time encompasses the sum of preprocessing, inference, and post-processing durations needed for the model to process an image with a batch size of 1.
3.3. Model Training Results and Visualization
The confusion matrix, a widely utilized instrument for assessing classification model performance, offers a straightforward visualization of how well the model performs across various classes. It offers a clear view of the model’s classification effectiveness by comparing the actual labels with the predicted labels. The normalized confusion matrices for the original YOLOv8s model and the enhanced model are depicted in
Figure 10a and
Figure 10b, respectively, with the rows corresponding to the actual classes and the columns to the predicted classes. The figures reveal that the enhanced model exhibits enhanced detection capabilities across all categories, particularly noteworthy in the detection of fruit stalks.
The boundary box loss, classification loss, and feature point loss during training and validation of the improved model, as well as its precision, recall, average precision at a 0.5 IoU threshold, and average precision at a 0.5:0.95 IoU threshold are shown in
Figure 11.
The visualization of the training results is shown in
Figure 12. We can clearly see that the recognition accuracy of the improved model has been significantly improved when dealing with small objects with severe occlusion. This improvement not only reflects the model’s ability to capture details but also reflects the model’s ability to adapt to complex scenes. In addition, the enhanced model demonstrates enhanced overall recognition accuracy, further validating the effectiveness of our refinement approach. Overall, these findings clearly illustrate the enhanced performance of the refined model.
3.4. Comparison Experiments with Different Object Detection Algorithms
To more intuitively demonstrate the performance of the improved YOLOv8 model, comparative experiments were conducted using Faster R-CNN, SSD, YOLOv5m, YOLOv5s, YOLOv7, YOLOv7-tiny, YOLOv8x, YOLOv8l, YOLOv8m, YOLOv8s, and YOLOv8n, all based on deep learning models, using the same raisin dataset. The comparative experimental results of different models are shown in
Table 2, and the experimental results of different models for different foreign matter classification accuracy are shown in
Table 3.
The detection performance of various models is illustrated in
Figure 13. The comparative experimental outcomes indicate that the refined YOLOv8 model exhibits superior performance in precision, recall, and the composite metric F1-score, particularly excelling in precision, with a marginal decrease compared to the YOLOv7 model. In terms of the average precision at a 0.5 IoU threshold, it is only slightly lower than the YOLOv8x model by 0.4 percentage points and is 4.1, 30.3, 1.9, 2.9, 0.1, 1.5, 0.2, 0.3, 1.5, and 2.7 percentage points higher than other models, respectively. In terms of the average precision at a 0.5:0.95 IoU threshold, it is lower than the larger YOLOv8 models and the YOLOv7 model. In terms of the number of parameters, FLOPs, and detection time, the improved model has a good performance. From the experimental results of different models for different foreign matter classification accuracy, the improved model has achieved a more significant improvement in detecting fruit stalk foreign matter, only slightly lower than the YOLOv8x model by 0.7 percentage points and is 9.0, 43.4, 5.0, 5.8, 2.8, 5.1, 1.7, 1.4, 3.1, and 6.9 percentage points higher than other models, respectively.
In summary, the enhanced model strikes an optimal harmony between detection accuracy and processing speed, exhibiting notable advancements in detecting various types of raisin foreign matter while introducing a minimal increment in the parameter count.
3.5. Ablation Experiments
To further verify the interaction effects between the improved modules and the optimization effects of the improved modules on the entire model, eight ablation experiments were set up for three groups of modules, as shown in
Table 4. These experiments systematically evaluate the impact of each module’s independent and combined work on the model’s performance.
When the MHSA module is introduced alone, the recall rate and the average precision at a 0.5 IoU threshold increase by 1.3% and 0.6%, respectively, while the precision rate and the average precision at higher IoU thresholds decrease by 0.3% and 1.0%, respectively. This indicates that the MHSA module enhances the model’s target recognition ability and the overall performance under more lenient criteria, but slightly increases the false detection rate. When the RFAConv module and the replacement with EIoU loss are introduced individually, the precision rate, recall rate, and average precision at both IoU thresholds all improve, indicating that the RFAConv module and EIoU loss make the model recognition more accurate and contribute significantly to enhancing the overall model effectiveness.
When both the MHSA module and RFAConv module are introduced simultaneously, compared with the baseline model and the model after introducing each module separately, the accuracy improves by 1.6%, 1.9%, and 1.2%, respectively, indicating that these two modules have a synergistic effect in improving the accuracy of the model in recognizing targets. In contrast to the baseline model, the integration of the MHSA module with EIoU loss outperforms the model employing the MHSA module alone or EIoU loss independently, the recall rate increases by 1.4%, 0.2%, and 0.9%, and the precision rate decreases by 0.6%, 0.3%, and 1.4%, respectively. This suggests that the MHSA module enhances the model’s recognition capabilities but marginally elevates the false detection rate, whereas EIoU loss amplifies the benefits of the MHSA module while preserving the average precision.
In contrast to the baseline model, the combined use of the RFAConv module with EIoU loss outperforms the model using the RFAConv module alone, and the model after replacing the loss with EIoU loss alone, the precision rate improves by 1.9%, 1.5%, and 1.1%, the recall rate slightly decreases, and the average precision at both IoU thresholds improves. This suggests that the integration of the RFAConv module with EIoU loss contributes significantly to enhancing the model’s overall effectiveness.
Finally, the improved model, under the synergistic effect of multiple improved modules, has a significant improvement on indicators such as precision rate, recall rate, and average precision at both IoU thresholds compared to the baseline model.
3.6. Comparison Experiments of Different Foreign Matters Combinations
To verify the detection performance of the improved model for single or multiple foreign matters, seven groups of comparative experiments were set up for four types of foreign matters, including combinations of raisins with fruit stems, raisins with branches, raisins with stones, raisins with plastic, raisins with fruit stems and branches, raisins with stones and plastic, and raisins with all types of foreign matters combined. The detection results are shown in
Figure 14. The outcomes of the experimentation demonstrate that the refined model showcases robust detection capabilities both in the detection of single foreign matters and in the detection of combinations of multiple foreign matters, but it is more difficult to detect foreign matters that are severely obscured.
4. Discussion
The automated detection of agricultural products is currently one of the more important research areas, and for raisins, a considerable number of researchers have already implemented automated detection and classification using traditional machine learning techniques. Abbasgolipour M. et al. [
22] engineered a machine vision-driven automatic sorting apparatus for raisins, comprising a conveyor belt, illumination module, control and processing unit, and a sorting mechanism. The system classifies raisins by initially segmenting them from the background in images and subsequently evaluating them based on hue, saturation, and intensity (HSI) color attributes, realizing a 93% classification accuracy. Mollazade K. et al. [
23] enhanced the preprocessing and segmentation of four raisin types—green, green with stems, black, and black with stems—leveraging image processing and data mining to extract a robust feature set of 44, including 36 color and 8 shape features. These data were fed into an Artificial Neural Network (ANN) with a 7-6-4 topology, achieving a 96.33% classification accuracy. Wang S. et al. [
24] introduced a classification approach that amalgamates morphological, color, and texture features from RGB images, yielding 74 features classified via five models, with the linear discriminant model leading in accuracy at 99%. Yu X. et al. [
25] proposed a classification strategy integrating color and texture features, employing linear discriminant analysis, SIMCA, and LS-SVM to construct models that identified a 95% accuracy rate in classifying four raisin types. Karimi N. et al. [
26] utilized four extraction techniques to distill 146 texture features, subsequently refined by principal component analysis (PCA), and applied ANN and SVM classifiers to differentiate between high-quality and substandard raisins, with SVM demonstrating superior accuracy at 92.71%. Çınar İ. et al. [
27] discerned seven morphological features and classified raisin varieties using logistic regression, multilayer perceptrons, and support vector machines, with SVM attaining the highest classification accuracy of 86.44%. Khojastehnazhand M. et al. [
28] advanced a texture-feature-based machine vision technique for bulk raisin classification, utilizing a Support Vector Machine with GLRM features to achieve 85.55% accuracy in quality classification and 69.78% in differentiating raisins from foreign matter such as sawdust and thistles. Sahin O. et al. [
29] compared six machine learning models in classifying Turkish raisins, with the Multi-Layer Perceptron outperforming others at 94.07% accuracy. MOHAMED T.M. et al. [
30] designed a raisin seed classification model, which employs feature normalization and principal component analysis for preprocessing, and then input the data into a fully connected neural network for the classification task, achieving an accuracy and F-Score exceeding 91%.
Current studies reveal that conventional machine learning techniques have reached an exceptional level of precision in raisin classification, surpassing industry grading benchmarks. However, these methods are mostly focused on the classification of the raisins themselves, neglecting the presence of endogenous and exogenous foreign matter during the drying, collection, and processing stages. Moreover, existing methods heavily rely on feature engineering, which often requires a significant amount of time and effort from researchers during the model-building process. Handling situations with material density and complex backgrounds is particularly challenging.
Deep learning is a new detection technology that has emerged in recent years, and a large number of researchers are currently using deep learning techniques to detect agricultural products. Parvathi S. et al. [
31] developed a deep learning model leveraging Faster R-CNN and ResNet-50 to identify the ripeness stages of coconuts against intricate backgrounds, attaining a model accuracy of 0.894 and an average processing velocity of 3.124 s per image. Wang Z. et al. [
32] put forth a tomato ripeness detection technique grounded in the MatDet model, an iteration of Faster R-CNN that employs ResNet-50 as its backbone, integrates RolAlign for precise bounding box delineation, and utilizes PANet for multi-scale feature fusion. This model excelled in complex environments with branch occlusions, fruit superpositions, and variable lighting, reaching an average accuracy of 96.14% and effectively mitigating challenges like occlusion, overlapping, and lighting variations. Wang Z. et al. [
33] also introduced a real-time apple stem/calyx recognition system founded on the YOLO-v5 algorithm, which through the optimization of hyperparameters, transfer learning, and the refinement of model parameters and weight volumes using detection head search algorithms and pruning techniques, achieves an overall reduction of approximately 71%, enabling a real-time detection rate of 25.51 frames per second on a CPU with an accuracy of 93.89%. Mathew M. P. et al. [
34] presented a disease detection methodology for sweet pepper plant leaves, capitalizing on the YOLOv5 model, which adeptly identified early bacterial spot infections with a compact model size, an average accuracy of 0.907, and an average processing speed of 20ms per image. Yang S. et al. [
35] proposed a strawberry ripeness detection strategy predicated on the LS-YOLOv8s model, an evolution of the YOLOv8 algorithm that assimilates the LW-Swin Transformers model and introduces novel random variables to govern contrast enhancement, thereby enhancing model performance. The LS-YOLOv8s model achieved a detection accuracy of 94.4% and a rate of 19.23 frames per second, with 51.93% of the parameters of the YOLOv8s model.
The current studies comprehensively showcase the applicability of deep learning in agricultural product detection, with the YOLO model exemplifying the efficacy of single-stage object detection approaches. Although its accuracy cannot be compared to that of traditional machine learning methods and two-stage object detection methods, it has significant advantages in terms of detection speed and the resources required, while also achieving product standard accuracy. In the detection of foreign matter in raisins, there are situations where materials and foreign matter overlap to some extent, and some foreign matter is very small, making detection even more challenging.
To address this, we propose an enhanced real-time detection approach leveraging an improved YOLOv8 model. This novel method integrates MHSA from BoTNet into YOLOv8’s backbone. In the model’s neck layer, selected C2f modules have been strategically replaced with RFAConv modules. The model also adopts an EIoU loss function in place of the original CIoU. Our experiments reveal that the refined YOLOv8 boasts a precision of 94.5%, a recall rate of 89.9%, and an F1-score of 0.921, with a mAP reaching 96.2% at the 0.5 IoU threshold and 81.5% across the 0.5–0.95 IoU range. For this model, comprising 13,177,692 parameters, the average time required for detecting each image on a GPU is 7.8 milliseconds. The model demonstrates good performance both in comparison with existing models and in the detection of samples composed of single or multiple foreign substances combined with raisins. Nonetheless, there remains scope for enhancing the research. First, although the model has shown some improvement in detecting raisins under conditions where they are obscured by foreign substances, there is still an issue of missed detection. Second, although the model has shown promising results in terms of accuracy and computational efficiency, further enhancements can be achieved through techniques such as pruning and knowledge distillation, which are also being explored in the domain of agricultural product detection. Third, the current model is only targeted at the seedless white variety of raisins from Turpan City, Xinjiang Uyghur Autonomous Region, China. In future research, the range of dried grape varieties can be expanded to increase the applicability of the model.
5. Conclusions
This study introduces a real-time detection approach for identifying foreign matter in mixed raisins, utilizing an enhanced version of YOLOv8. The proposed method integrates BoTNet’s multi-head self-attention (MHSA) into YOLOv8’s backbone, substitutes the neck layer’s Conv module with RFAConv, switches from the original CIoU loss to EIoU loss, and uses a new raisin foreign matter dataset for training. After experiments, the F1-score and mAP0.5 of the improved model increased by 1.21% and 1.48%, respectively, compared to the original model, and its comprehensive performance is close to that of the YOLOv8m model with 96.1% more parameters.
Comparing the average precision of the improved and original models for different categories, although the detection performance of this method has improved in various categories, there are still problems of being unable to detect or misjudge small or severely occluded foreign matter, especially in the detection of fruit stalks and branches. In comparison to various deep learning-based object detection algorithms, this approach delivers excellent detection outcomes with reduced parameter count and enhanced speed, striking an optimal balance between accuracy and efficiency. It is hardware-efficient and demonstrates robust practical utility.
Using this method for rapid detection of foreign matter in raisins and combining it with a removal platform can greatly reduce labor costs, reduce hardware costs, and promote the intelligent development of the raisin industry, which has a positive significance for the promotion of agricultural products. Currently, this method still has shortcomings in detecting dense small objects and occluded objects. In our ongoing efforts, we plan to delve into refining and streamlining the model for implementation on mobile and embedded systems.