1. Introduction
Sichuan pepper (Zanthoxylum bungeanum) is a spice and traditional Chinese medicine originating from Sichuan Province, China, renowned for its unique flavor and medicinal value [
1,
2,
3]. China is the largest producer of Sichuan pepper globally, making it an important specialty crop with significant economic value [
4]. The quality of Sichuan pepper can be influenced by environmental factors such as the growing region, climate, and temperature, leading to varying quality levels [
5]. Temperature changes can affect the ripening of Sichuan peppers; under high temperatures, the fruit may ripen too quickly, affecting its flavor and aroma. Excessive rainfall increases the rate of fruit cracking, thereby reducing the quality and market value of the Sichuan peppers. Therefore, grading fresh Sichuan peppercorns can meet the diverse needs of different markets, better ensuring quality control and enabling producers to maximize returns and resource utilization. Grading fresh Sichuan peppercorns also improves the efficiency and quality of subsequent secondary processing, which has a significant economic impact. Adopting different processing methods according to the quality of the Sichuan peppercorns can reduce post-harvest losses, decrease the rate of defective products, and improve overall product quality. The enhancement in efficiency and quality directly relates to increased profitability for producers. Graded Sichuan peppercorns are better positioned to meet the varying demands of the market, promote product diversification, and enhance added value.
To promote intelligent, sustainable, and efficient agricultural development, smart agriculture has become a key direction for future agricultural development. Agricultural robots are essential for processes such as planting, irrigation, and harvesting, with machine vision being the core component of these robots [
6]. With the rapid advancement of vision technology, deep learning-based vision segmentation techniques are widely used in agriculture [
7,
8,
9]. Agricultural robots use visual systems to analyze crops from multiple angles and dimensions, performing tasks such as sorting, harvesting, and picking. However, cluster crops like Sichuan pepper, grapes, and lychees pose challenges for agricultural machine vision systems due to their clustered growth, overlapping, and complex backgrounds [
10,
11]. Peng et al. [
12] constructed the RDf-DeepLabV3+ network to enhance the segmentation performance of lychees, achieving a mIoU of 77% under complex background conditions. Wang et al. [
13] designed the DualSeg network, based on CNN and transformer, for the segmentation of grape clusters and branches, achieving an mIoU of 82.7%. Xu et al. [
14] proposed an improved Mask R-CNN network for the visual system of agricultural robots, achieving a recognition accuracy of 93.76% for cherry tomatoes. Wang et al. [
15] developed a perception model for the detection and segmentation of Sichuan pepper and branches, achieving an
of 90.1% with an mIoU of 84.2% and 89.9% for Sichuan pepper and branches, respectively, aiding in pepper harvesting. The above studies focus only on the identification and segmentation of crops during the growth process, without exploring the post-harvest handling of crops. This research will further investigate the post-harvest processing of Sichuan peppers and analyze their quality to support the subsequent selection of high-quality Sichuan peppercorns.
To enhance crop economic value, it is crucial to accurately grade crop quality quickly after analysis. Machine vision plays a vital role in crop grading [
16,
17]. Ji et al. [
18] proposed a multi-view method for grading apple quality, using an improved YOLOv5s algorithm to detect and grade defects from multi-angle images. Deng et al. [
19] developed an online carrot grading system based on machine vision and deep learning, using the CDDNet network to detect defects and MBR fitting and convex polygon approximation for quality grading. TU et al. [
20] used image recognition software and machine learning to separate high-quality pepper seeds from inferior ones, using eight features such as color, width, length, projection area, and single-core density for prediction with a multilayer perceptron. Wang et al. [
21] graded dried Sichuan pepper using image processing and BP neural networks, achieving a classification accuracy of 79.53% to 98.04% by extracting RGB and HIS values from pepper images. Although deep learning performs well in image processing, it also introduces challenges such as efficiency issues, complex feature extraction, high computational complexity, and a large number of network parameters. On the other hand, while machine learning algorithms offer good processing speed, they perform poorly with high-dimensional data-like images [
22]. Moreover, they are less effective than deep learning in handling large-scale data, making it difficult to meet the real-time grading requirements for crop quality.
In actual sorting processes, model performance and multi-dimensional image analysis are necessary to ensure sorting algorithm accuracy. Currently, there is a lack of research on quality grading for fresh Sichuan pepper. Traditional manual sorting is inefficient, inaccurate, and labor intensive. As a specialty economic crop with high economic and medicinal value, integrating emerging computer and AI technologies into the Sichuan pepper industry can better support its development. Therefore, developing an algorithm for sorting high-quality Sichuan pepper to efficiently, quickly, and accurately grade its quality is essential. This paper addresses the challenges of deep learning networks’ computational load, high parameter count, and complex feature extraction by enhancing model accuracy with spatial, frequency, and channel domain feature information. Additionally, the network is lightweight to suit real-time sorting requirements in agricultural scenarios, proposing a new algorithm for sorting high-quality fresh Sichuan pepper. The main contributions of this paper are as follows:
Proposed a multi-scale frequency domain feature fusion module (MSF3M) to capture multi-scale frequency domain image features.
Proposed a multi-scale dual domain feature fusion module (MS-DFFM) to capture multi-scale channel and spatial domain features.
Developed a high-quality fresh Sichuan pepper sorting algorithm based on the average local pixel value difference method.
The rest of the paper is organized as follows:
Section 2 details the methods for Sichuan pepper dataset collection and construction, the MultiDomain YOLOv8 model network, the MSF3M and MS-DFFM module construction, and the high-quality fresh Sichuan pepper sorting algorithm implementation.
Section 3 presents extensive comparative and ablation experiments to evaluate the proposed approach, demonstrating its scientific validity and rationality.
Section 4 discusses the limitations of the proposed algorithm.
Section 5 provides a general summary of the study.
2. Materials and Methods
2.1. Overview of the Proposed Fresh Sichuan Pepper Quality Classification System
The overall process of the proposed fresh Sichuan pepper quality classification system is illustrated in
Figure 1, consisting of three main stages.
In the first stage, as shown in
Figure 1b, images of mature and semi-mature Sichuan peppercorns are segmented and labeled, as depicted in
Figure 1a. The MultiDomain-YOLOv8-seg model is used to learn the features of individual peppercorns.
In the second stage, as shown in
Figure 1d, images of pepper clusters from the front and back views are collected, as illustrated in
Figure 1c. The best-trained MultiDomain-YOLOv8-seg model is then applied for prediction, and the coordinates of the predicted peppercorns are used to crop single peppercorn images from the original images, as shown in
Figure 1e.
In the third stage, as shown in
Figure 1f, the segmented dataset is first input into the MultiDomain-YOLOv8-cls classification network for feature learning, aiming to classify mature and semi-mature peppercorns. Next, the average local pixel value difference of the predicted mature peppercorns is calculated to verify the color uniformity. Finally, by analyzing the sparsity and proportion of black pixels in the images, high-quality peppercorns are selected for future classification of fresh Sichuan pepper quality. The specific algorithm flow is represented in the generated image, as shown in
Figure 1g.
2.2. Data Acquisition and Annotation
To better train the model, we collected a dataset of Sichuan peppers with varying degrees of ripeness in Hanyuan County, Sichuan Province, China. To meet the grading requirements, we mainly collected semi-mature and mature fresh Sichuan peppers from Hanyuan and labeled the collected data according to the segmentation task requirements. The peppers were photographed during both morning and afternoon sessions on a single day. Since the ripeness of the Sichuan peppers is influenced by altitude and temperature, we captured images of peppers with different ripeness levels from both the mountaintop and the foothills at the Hanyuan Sichuan pepper plantation. A standard smartphone camera was used to capture the images.
During the shooting process, individual pepper clusters were photographed from a front-lit direction at a distance of approximately 5–10 cm, using consistent and stable artificial lighting to avoid the impact of natural light on image quality. A total of 1942 images were collected and labeled. The dataset was split in a 9:1 ratio, with 1624 images as the training set, 181 as the validation set, and 100 images selected as the test set to evaluate the segmentation performance of the proposed algorithm.
The collected Sichuan pepper images were labeled using the AnyLabeling tool, which integrates the segment anything model (SAM). This open-source labeling platform supports multiple model integrations and can automatically generate precise annotation data. The Sichuan peppers in the images were labeled at the granule level, and the corresponding collected images and labeling results are shown in
Figure 2.
To validate the performance of the high-quality Sichuan pepper classification algorithm proposed in this study, we collected 520 images of post-harvest Sichuan pepper clusters, including 246 clusters of semi-mature peppers and 274 clusters of mature peppers. Due to the influence of environmental factors such as light, rainfall, and temperature, there are significant differences in the ripeness of the front and back sides of the Sichuan peppers. These differences may affect the model’s ability to accurately assess the quality of the peppers. To capture these potential variations, images of both the front and back sides of the pepper clusters were taken during the data collection process. This approach allows the model to acquire feature information from different angles, leading to more accurate identification and classification of the ripeness and quality of the peppers.
This multi-angle data collection strategy not only increases the diversity of the dataset but also enhances the model’s classification accuracy and generalization ability, ensuring the effective selection of high-quality Sichuan peppers. During the data collection process, all images were captured under the same conditions and lighting setup. The corresponding collected images are shown in
Figure 3.
2.3. Sichuan Pepper Instance Segmentation Model: MultiDomain YOLOv8-seg
This section provides a detailed explanation of the proposed Sichuan pepper instance segmentation model, MultiDomain YOLOv8-seg. YOLOv8 is flexibly designed and easy to integrate, offering outstanding segmentation and classification performance. It maintains high accuracy while achieving faster inference speed.
The segmentation model is designed based on the YOLOv8 segmentation framework, introducing a multi-scale frequency domain feature fusion module (MSF3M) and a multi-scale dual-domain feature extraction module (MS-D2F2M) to integrate multi-domain and multi-scale image feature information. The model combines feature extraction in the frequency domain, spatial domain, and channel domain, leveraging the advantages of all three to enhance the model’s feature representation capability and robustness. The specific design includes key components such as an attention module, multi-scale feature information fusion, and an up-sampling module. It also incorporates programmable gradient information to achieve a lightweight deep neural network with high accuracy and fast inference speed.
In the backbone network, the RepNCSPELAN4 and Adown modules from YOLOv9 [
23] are introduced. The architecture of the RepNCSPELAN4 module is shown in
Figure 4, which is used to reduce the network parameter count while improving accuracy. In the shallow network, where the receptive field is small, it can extract low-level feature information such as edges, textures, and colors. The RepNCSPELAN4 module efficiently aggregates multi-scale, multi-layer detailed feature information in the shallow network. As the network depth increases, semantic feature information becomes richer, resulting in a significant increase in computational load. Therefore, using RepNCSPELAN4 would cause a substantial increase in computation and parameters. To address this, the C2f module with an integrated SE attention mechanism [
24] is used to focus on key feature information and accelerate network inference operations. Additionally, a multi-scale frequency domain feature fusion module (MSF3M) is proposed in the deep network for extracting multi-scale frequency domain feature information, capturing different frequency components in the input image to enhance the model’s feature extraction performance and accuracy. Finally, the SPPELAN module replaces the SPPF module in YOLOv8, combining the advantages of SPP and ELAN, further enhancing feature extraction capability through an efficient layer aggregation strategy.
In the neck part, traditional spatial up-sampling operators rely heavily on local pixel focus and fail to explore global dependencies. Therefore, a deep Fourier up-sampling operator [
25] is introduced to address the issue of CNN networks in deep networks being unable to capture long-distance feature information. Additionally, a multi-scale dual-domain feature fusion module (MS-D2F2M) is proposed, which extracts feature information from different scales in the spatial and channel domains to improve network accuracy. The proposed network integrates multi-scale spatial, frequency, and channel domain features, performs feature fusion processing, and introduces lightweight modules and attention modules to further accelerate network inference, significantly enhancing segmentation accuracy and reducing network computational complexity.
2.4. Multi-Scale Frequency Domain Feature Fusion Module (MSF3M)
This section provides a detailed explanation of the multi-scale frequency domain feature fusion module (MSF3M), hereafter referred to as MSF3M. Its structure is shown in
Figure 5. This module is designed for multi-scale frequency domain feature extraction in deep networks. Deep networks often face challenges such as gradient vanishing, overfitting, and high computational complexity during image feature extraction [
26]. Frequency domain feature extraction offers advantages over spatial domain feature extraction in terms of global information capture, translation invariance, and multi-scale feature fusion. Therefore, the MSF3M module is proposed for multi-scale feature extraction in deep networks. The MSF3M module consists of four main parts:
Part One: In the first part of the MSF3M module, the input image undergoes a fast Fourier transform (FFT) to convert the image from the spatial domain to the frequency domain, resulting in a complex representation with magnitude and phase components. The magnitude primarily describes the intensity of frequency components, while the phase describes the phase shift of frequency components. Phase information plays a crucial role in image feature extraction and reconstruction, while magnitude information contains most of the image’s texture and structure.
Part Two: In the second part of the MSF3M module, multi-scale feature extraction is performed on the magnitude and phase information. The feature extraction process for magnitude and phase is consistent; this part uses magnitude as an example. First, the magnitude (Magnitude) undergoes feature extraction using a 5 × 5 convolution. Then, the feature-extracted image is subjected to three sets of layered convolutions: 1 × 7 and 7 × 1, 1 × 11 and 11 × 1, and 1 × 21 and 21 × 1. This results in multi-scale feature information for the magnitude. Finally, a 1 × 1 convolution layer is applied to increase non-linearity and achieve cross-channel information fusion, thereby obtaining more feature information.
Part Three: In the third part of the MSF3M module, the multi-scale convolutions’ outputs, (magnitude) and (phase), are used for frequency domain reconstruction. The real part of the complex matrix is obtained by multiplying by the cosine of . Similarly, the imaginary part of the complex matrix is obtained by multiplying by the sine of . The real and imaginary parts are then combined to form the complex matrix in the frequency domain.
Part Four: In the fourth part of the MSF3M module, the complex matrix obtained from the third part undergoes an inverse fast Fourier transform (IFFT) to convert the image back from the frequency domain to the spatial domain. The output is then multiplied by the original input to retain important features from the original input image, thereby enhancing the model’s performance.
2.4.1. Multi-Scale Frequency Domain Feature Fusion Module Part I
MSF3M Part One: After applying the fast Fourier transform (FFT) to the input image, the image is transformed from the spatial domain to the frequency domain. The frequency domain image represents the image information in the frequency domain, where the low-frequency components are located in the central region of the frequency domain image, preserving the overall structure and primary feature information of the image. The high-frequency components are located at the edges of the frequency domain image, preserving the details and edge information of the image. Furthermore, the magnitude and phase of the frequency domain signal are extracted. The magnitude retains the main feature structure of the image, while the phase retains the spatial information of the image. Therefore, multi-scale feature extraction is subsequently performed on the magnitude and phase information.
In this study, the FFT algorithm is used to transform the input features from the spatial domain to the frequency domain. The fast Fourier transform is an efficient algorithm for computing the discrete Fourier transform (DFT). It decomposes an N-point DFT into two
-point DFTs, using a recursive approach for computation. The time complexity of the FFT is only
, significantly reducing the computational load compared to the
complexity of the DFT, thus accelerating network inference and analysis. The corresponding fast Fourier transform formula can be expressed as:
where
represents the feature map in the frequency domain,
represents the feature map in the spatial domain,
is the imaginary unit,
and
denote the dimensions of the input signal.
To obtain the magnitude and phase of the frequency domain feature information, the complex matrix in the frequency domain is further processed. Using Euler’s formula, the real part matrix
and the imaginary part matrix
are derived from the complex matrix. Furthermore, the formulas for the magnitude (2) and phase (3) in the frequency domain can be expressed as:
where
denotes the magnitude, and
denotes the phase. The transformation from the spatial domain to the frequency domain, including the magnitude image and phase image, is illustrated in
Figure 6.
2.4.2. Multi-Scale Frequency Domain Feature Fusion Module Part II
MSF3M Part Two: Multi-scale feature extraction is performed on the magnitude and phase information in the frequency domain image to further obtain the image’s feature information.
To further extract features from the magnitude and phase, a 5 × 5 convolution operation is first applied to both the magnitude and phase to extract overall features. The resulting images are then processed through three sets of layered convolutions with different kernel sizes: 1 × 7 and 7 × 1, 1 × 11 and 11 × 1, and 1 × 21 and 21 × 1, to capture features at different scales. After feature extraction, the multi-scale feature information is aggregated. Finally, a 1 × 1 convolution is applied to reduce dimensionality, decreasing computational costs, and enhancing the network’s expressive capability. The final outputs are the multi-scale feature-extracted magnitude information,
, and phase information,
. The formula for the multi-scale convolution can be expressed as:
where
represents the input image features and
represents the convolution kernel.
2.4.3. Multi-Scale Frequency Domain Feature Fusion Module Part III
MSF3M Part Three: Reconstruction of the frequency domain feature information involves reconstructing the magnitude information, , and phase information, , into a complex matrix in the frequency domain.
The magnitude information,
, and phase information,
, obtained after multi-scale feature extraction are reconstructed into the real part,
, and the imaginary part,
, of the complex matrix. The relevant calculation formulas are expressed as:
After obtaining the real and imaginary parts of the complex matrix, the real part matrix and the imaginary part matrix are combined to form the complex matrix
, as represented below:
2.4.4. Multi-Scale Frequency Domain Feature Fusion Module Part IV
MSF3M Part Four: The frequency domain feature map is transformed back to the spatial domain using the inverse fast Fourier transform (IFFT). The spatial domain image is then multiplied with the original input feature map to further enhance the image’s feature information.
To transform the frequency domain feature map back to the spatial domain, the inverse fast Fourier transform is applied. This process can be represented as:
At this stage, the multi-scale feature information in the frequency domain has been extracted. However, transforming the frequency domain image back to the spatial domain may introduce information loss or errors. To mitigate this, further feature enhancement is performed by multiplying the original spatial domain image with the transformed image. This operation preserves the image’s feature information to a large extent, resulting in an enhanced feature map. The corresponding calculation can be expressed as:
As illustrated in
Figure 7, the feature heatmaps after applying the MSF3M module exhibit more uniform activation areas, and more dispersed feature information, and retain more detailed information with less noise, resulting in higher-quality feature extraction from the images.
2.5. Multi-Scale Dual-Domain Feature Fusion Module (MS-DFFM)
This section introduces the multi-scale dual-domain feature fusion module (MS-DFFM), hereafter referred to as MS-DFFM. Its architecture is shown in
Figure 8. This module integrates channel feature information and multi-scale spatial feature information, ultimately outputting image features enhanced by dual attention mechanisms. The module consists of two main parts: the channel attention mechanism and the multi-scale attention mechanism.
First, the SE (squeeze-and-excitation) channel attention mechanism is used to extract channel feature information from the image. The input tensor is first passed through an adaptive pooling layer and a 1 × 1 convolution. Then, it undergoes a nonlinear transformation via the HSigmoid activation function. The resulting output is multiplied by the input tensor to form the channel attention mechanism.
Next, the vector containing channel feature information is fed into the multi-scale convolution attention module [
27], propagating along the multi-scale spatial dimensions. Initially, it goes through a 1 × 1 convolution for feature transformation, followed by a nonlinear transformation using the GELU activation function. Meanwhile, the input feature data enters the multi-scale convolution attention mechanism for multi-scale spatial feature extraction. This begins with a 5 × 5 convolution for feature transformation, followed by feature extraction using four different scales of convolution: 1 × 3 and 3 × 1, 1 × 7 and 7 × 1, 1 × 11 and 11 × 1, and 1 × 21 and 21 × 1. Finally, the multi-scale feature information is dimensionally reduced through a 1 × 1 convolution. This reduced feature map is then added to the original input via a residual connection, resulting in features enhanced by dual attention mechanisms. This enhancement further increases the network’s accuracy and performance.
2.6. Deep Fourier Up-Sampling Module
Traditional up-sampling operators often heavily rely on local pixel attention [
28] and typically use fixed interpolation strategies, making it difficult to adjust strategies based on the input image content. They also lack the ability to model contextual information, which can result in poor performance for certain tasks such as super-resolution reconstruction and image generation. Deep neural networks, particularly convolutional neural networks (CNNs), perform convolutions locally, operating only on local input data and lacking the ability to extract contextual information. Even though methods like varying convolution kernel sizes and pyramid pooling are used to capture multi-scale features, they struggle to address the issue of up-sampling operators relying on local pixels.
In this study, a deep Fourier up-sampling module is introduced to model global characteristics and extract long-range dependencies. After applying the fast Fourier transform (FFT), the frequency domain information exhibits global properties. Processing this frequency domain information is more suitable for extracting global characteristic information.
The deep Fourier up-sampling in this study addresses the up-sampling problem by cyclically padding the amplitude and phase of the frequency domain signals. This module uses the FFT to obtain the amplitude and phase components of the frequency domain and then cyclically pads these two components in the H and W dimensions. Feature extraction is performed using a convolution module, and finally, the image is transformed back to the spatial domain using the inverse fast Fourier transform (IFFT). The padding process is illustrated in
Figure 9.
2.7. A Method for Quality Sorting of Fresh Sichuan Pepper
This section will discuss the quality sorting of freshly picked Sichuan peppercorns, primarily using visual assessment to preliminarily grade the color quality of the peppercorns.
Table 1 shows the quality grading standards for Hanyuan Sichuan peppercorns [
29].
In this study, the quality sorting of Sichuan pepper is primarily based on the color of the peppers, focusing on two aspects: the ripeness of the peppers and the color uniformity of individual pepper grains. Detailed explanations of these two aspects are provided below.
To effectively classify mature and semi-mature Sichuan peppers after harvesting, we photographed the harvested peppers in the same scene. Due to various environmental factors during growth, peppers may not ripen uniformly. Therefore, we photographed both the front and back of the pepper clusters to ensure classification accuracy. The images were then input into the MultiDomain YOLOv8-seg model for segmentation, and the predicted segmentation coordinates were used to crop the original images. The cropped individual pepper grains were saved. Based on their appearance, the peppers were saved into folders for mature and semi-mature peppers. The dataset includes 2246 images of mature peppers and 1962 images of semi-mature peppers, which were divided into training and testing sets in an 8:2 ratio, as shown in
Figure 10.
As shown in
Figure 10, there are significant differences in ripeness and color between the front and back of the peppers. Therefore, assessing the quality from a single angle would be inaccurate. This study measures the peppers from two angles to validate the proposed algorithm’s reasonableness and accuracy.
2.8. Sichuan Pepper Classification Model: MultiDomain YOLOv8-cls
To better validate the performance of the proposed multi-scale frequency domain feature fusion module (MSF3M) and multi-scale dual domain feature fusion module (MS-DFFM), and to rapidly classify mature and semi-mature Sichuan peppercorns, this study proposes the MultiDomain YOLOv8-cls model. This model is designed based on the YOLOv8n-cls model, with the overall design structure shown in
Figure 11.
In the network structure design, the MS-DFFM module is added to the shallow layers to extract multi-scale channel and spatial feature information. The MSF3M module is integrated into the deeper layers to focus on long-range dependencies by extracting multi-scale frequency domain features, retaining the rich feature information of the image. This significantly enhances the network’s accuracy and performance. The proposed network is lightweight, with low computational complexity and a small number of parameters, making it highly suitable for cost-sensitive smart agriculture scenarios where memory and computational resources are limited. The proposed classification network can better fit the requirements of smart agriculture, promoting agricultural intelligence.
High-Quality Fresh Sichuan Peppercorn Sorting Based on Average Local Pixel Value Difference
In this section, we will elaborate on the quality assessment and sorting process for mature Sichuan peppercorns post-classification. High-quality fresh peppercorns should have characteristics such as full, vibrant colors and uniformity. We propose a high-quality peppercorn sorting algorithm based on the average local pixel value difference. This algorithm consists of two main parts:
Average Local Pixel Value Difference Method: This method determines regions in the image where the color or grayscale values change minimally by calculating the average local pixel value difference. Regions with small differences exhibit relatively uniform color or grayscale without distinct boundaries or textures. This method effectively distinguishes the flat color areas in peppercorn images, further determining color uniformity. The algorithm flow is illustrated in
Figure 12.
The algorithm is divided into four parts:
- (1)
Image Reading and Grayscale Conversion: Read the peppercorn image and convert it to a grayscale image.
- (2)
Application of Convolution Filter: Calculate the average value of each pixel and its neighboring pixels using a 3 × 3 convolution kernel, and then compute the difference between each pixel and the average value of its neighbors (vALD).
- (3)
Calculation of Global Average Pixel Value and Difference: Compute the global average pixel value of the image and the difference between each pixel and the global average.
- (4)
Flat Region Marking: Mark the flat color regions of the image based on the previous calculations. Pixels with a difference lower than 10% of the global average pixel difference are marked white to produce the flat region labeled image.
Flat Region Labeled Image Assessment: To determine the color uniformity of the peppercorns, the flat region labeled images generated by the average local pixel value difference method are further classified. This assessment involves evaluating the sparsity and proportion of black pixels in the image, as detailed in
Figure 13.
This algorithm is divided into three parts:
- (1)
Black Pixel Proportion Calculation: Calculate the proportion of black pixels in the entire image, representing the proportion of non-flat regions.
- (2)
Local Sparsity Detection: In the generated flat region labeled image, black pixels might be scattered throughout. If the proportion of black pixels in the entire image is low, the image is also considered non-flat. A sliding window is used to calculate the black pixel proportion in each window.
- (3)
Global Sparsity Detection: Calculate the global black pixel proportion and compare it with a given global threshold. Images with a black pixel proportion lower than the threshold are classified as high-quality peppercorns.
By utilizing these methods, we can effectively screen and identify high-quality Sichuan peppercorns based on their visual characteristics, particularly focusing on color uniformity and maturity.
2.9. Implementation Details
This section details the experimental procedures for both the MultiDomain YOLOv8-seg and MultiDomain YOLOv8-cls models. No pre-trained weights were used during model training, as this study introduces significant modifications to the YOLOv8 model, including adjustments to the network structure and modules, leading to notable differences from the original model. Additionally, the dataset used in this study differs considerably from those commonly used in pre-training models (e.g., models trained on COCO). Training from scratch allows the model to better adapt to new task requirements, ensuring that it fully learns relevant features while avoiding irrelevant features or noise introduced by pre-trained weights.
To validate the advantages of the improved models and ensure experimental fairness, both the original and improved models were trained from scratch under identical conditions. Although this approach increases training time and data requirements, it helps accurately evaluate the performance of the improved models and ensures the reliability of the comparative experiments. The specific hyperparameter settings for the model during training are detailed in
Table 2.
2.9.1. Implementation Details of the Segmentation Model
The constructed pepper dataset contains 1942 images, divided into training (1624 images), validation (181 images), and test (100 images) sets with a 9:1 ratio. Segmentation experiments are conducted on a Windows server equipped with an NVIDIA Quadro RTX 5000 16 GB GPU, an Intel(R) Core(TM) i9-10900K CPU @ 3.70 GHz, 128 GB RAM, and Windows Server 2019 64-bit OS.
2.9.2. Implementation Details of the Classification Model
The fresh Sichuan peppercorn dataset comprises 4000 images, divided into training (2179 images), validation (234 images), and test (500 images) sets with a 9:1 ratio. Classification experiments are conducted on a Windows computer equipped with an NVIDIA GeForce RTX 2070 SUPER 8 GB GPU, an AMD Ryzen 7 3700X 8-Core Processor CPU @ 3.70 GHz, 16 GB RAM, and Windows 11 Professional 64-bit OS.
2.10. Evaluation Metrics
In this experiment, the performance of the models is evaluated using six metrics: parameter count, GFLOPs, accuracy,
, recall, and FPS. GFLOPs (giga floating point operations per second) is an important metric for evaluating model complexity. It indicates the speed and computational capability of a neural network model by measuring the number of floating-point operations the model can perform per second.
where
represents the number of input channels,
represents the number of output channels,
represents the convolution kernel size,
represents the width of the input image, and
represents the height of the input image.
Here,
indicates the proportion of true positive samples among all samples predicted as positive.
indicates the proportion of positive samples correctly identified as positive among all actual positive samples.
represents the number of true positive cases,
represents the number of false-positive cases, and
represents the number of false-negative cases.
(mean average precision) represents the average precision at an IoU threshold of 0.5 for all classes, where
is the number of classes, and
is the average precision for class
.
To better evaluate the application of the models in smart agriculture scenarios, FPS (frames per second) is included as an evaluation metric in the classification model to assess the real-time inference performance.
In this study, the segmentation model is evaluated using accuracy, recall, , GFLOPs, and parameter count. The classification model is evaluated using accuracy, GFLOPs, parameter count, and .
4. Discussion
4.1. Analysis of the Impact of MSF3M and MS-DFFM Modules on Network Performance
This section will focus on analyzing why the individual addition of the multi-scale frequency domain feature fusion module (MSF3M) and the multi-scale dual-domain feature fusion module (MS-DFFM) to a classification network leads to a decline in model performance and explore the reasons behind the significant improvement in performance when both MSF3M and MS-DFFM modules are included simultaneously.
4.1.1. Why Adding MSF3M and MS-DFFM Modules Separately Leads to a Decline in Network Performance
This section will specifically examine why adding either the MSF3M or MS-DFFM module individually leads to a decline in model performance within a classification network. For the dataset of fresh Sichuan peppercorns constructed in this study, the feature extraction process focuses more on local details, textures, and color features of the peppercorns, with relatively less emphasis on global information. The classification model network has a relatively shallow depth, while the MS-DFFM module primarily extracts multi-scale information from the images. The use of large-scale convolution introduces more global feature information, which, although beneficial for local feature extraction, can also introduce additional noise or lead to overfitting, resulting in a slight decrease in model performance. On the other hand, the MSF3M module mainly performs global feature extraction in the frequency domain, with relatively less emphasis on local information. The lack of these detailed local features can lead to a decline in model accuracy.
4.1.2. Why Adding Both MSF3M and MS-DFFM Modules Simultaneously Improves Network Performance
This section will analyze why the simultaneous addition of MSF3M and MS-DFFM modules leads to a significant improvement in model performance. The introduction of both modules provides complementary feature extraction methods. MSF3M extracts global information in the frequency domain, compensating for the lack of global feature information in the spatial domain. On the other hand, the MS-DFFM module focuses on local spatial and channel features, capturing finer-grained details. By complementing global and local feature information, the model greatly enriches the image features, enhances feature representation capabilities, reduces the risk of overfitting, and gains a more comprehensive understanding of the input data, leading to a substantial improvement in network performance. The generated features are illustrated in
Figure 19.
As shown in
Figure 19, after incorporating both the MSF3M and MS-DFFM modules, the feature maps exhibit a more balanced distribution. This approach retains the detail-capturing ability of the MSF3M module while enhancing the salient feature extraction capability of the MS-DFFM module. Consequently, the simultaneous use of these two modules significantly enhances the network’s feature extraction capacity.
Figure 20 presents the results of visual feature extraction using Grad-CAM for the proposed MultiDomain YOLOv8 model and the YOLOv8 model. After feature extraction on Sichuan pepper, mature peppercorns, and semi-mature peppercorns through the MS-DFFM and MSF3M modules, the generated heatmaps demonstrate a more refined feature extraction capability, better separating pepper clusters from the background. The visualization shows clearer contours and significant color variations, with stronger contrast in the highlighted areas, which more effectively highlights the key features of the peppercorns and reduces noise interference. Compared to the baseline network, the MultiDomain YOLOv8 model more clearly captures the texture details and differences in various regions of the peppercorns. Experimental results indicate that the MS-DFFM and MSF3M modules proposed in this study significantly improve feature extraction accuracy, background separation, response to different maturity levels, and anti-interference capability in the tasks of Sichuan pepper segmentation and maturity classification, demonstrating their effectiveness and reliability compared to the YOLOv8 model.
4.2. Limitations of the Quality Sorting Algorithm for Fresh Sichuan Pepper
In practical agricultural applications, various factors can influence the high-quality sorting of Sichuan peppercorns. The following analysis examines how these factors affect the algorithm:
Lighting Effects: In real sorting conditions, variations in lighting can significantly impact the quality of peppercorn images. Especially in outdoor environments, lighting conditions change with time, weather, and location, leading to noticeable variations in image brightness and contrast. Such inconsistencies can cause a decrease in model performance during the training and testing phases of deep learning algorithms. In this study, the model was trained on images captured under fixed lighting conditions. However, in practical scenarios with different lighting conditions, the model’s recognition accuracy may be compromised. Moreover, shadows can cause anomalies in local pixel values, leading to errors in methods like average local pixel value difference and flat region marking, which may result in more incorrect flat region labels, affecting the algorithm’s decision-making.
Impact of Classification Algorithm: When classifying mature and semi-mature peppercorns, misclassification may occur, where some semi-mature peppercorns are incorrectly categorized as mature. Semi-mature peppercorns may have a mix of red and green colors, and if both colors are uniformly distributed, the average local pixel value difference method might mark them as flat regions, thus influencing the algorithm’s judgment.
Image Acquisition Impact: During the actual image acquisition process, issues such as occlusion between peppercorn clusters and the angle of shooting can result in incomplete assessments of peppercorn quality. Although this algorithm captures images from both the front and back of individual peppercorn clusters, blind spots may still exist. Occlusion may prevent some semi-mature peppercorns from being captured in the image, and subsequent cropping may fail to include all the peppercorns in the cluster, thereby affecting the algorithm’s ability to accurately assess the quality of the peppercorn cluster.
Impact of Image Noise: When collecting peppercorn images under uncontrolled conditions, the images may contain various types of noise, such as environmental noise, sensor noise, and motion noise. These noises can hinder the model’s ability to extract key features from the images, leading to a decrease in the accuracy of classification and segmentation algorithms. Especially when the signal-to-noise ratio is low, the algorithm may struggle to distinguish between the target and the background, making feature extraction more challenging and resulting in reduced model performance.
4.3. Future Prospects and Agricultural Applications
The MultiDomain-YOLOv8 model, which integrates multi-domain and multi-scale features, has demonstrated outstanding performance in Sichuan pepper segmentation and grading tasks by more comprehensively and accurately extracting image features. These advantages suggest that this algorithm holds significant potential for broader agricultural applications beyond Sichuan pepper grading. Specific potential applications include:
Crop Health Detection: The algorithm’s ability to integrate multi-scale features enables precise identification of various stages of crop diseases and pests, such as leaf spot detection and pest detection on leaves. By combining multi-domain image features, the algorithm can better capture subtle differences in diseased areas. This capability allows for timely intervention, helping farmers to control pests and diseases effectively, thereby reducing losses and ensuring the healthy growth of crops.
Fruit Ripeness Grading: Beyond Sichuan pepper, the MultiDomain-YOLOv8 model can be applied to ripeness detection of other fruits, such as apples, citrus, and grapes. Different stages of fruit ripeness exhibit significant differences in color, texture, and shape. The integration of multi-domain and multi-scale features enhances the model’s ability to identify detailed characteristics, accurately distinguishing between ripe and unripe fruits, thereby improving sorting and grading efficiency within the agricultural supply chain.
Applications in Smart Agricultural Machinery: The strong feature extraction capability of MultiDomain-YOLOv8 can provide precise visual guidance for automated harvesting robots. During the harvesting process, the model can accurately identify ripe fruits or monitor crop growth, facilitating intelligent harvesting and management.
In summary, the multi-scale and multi-domain feature extraction capabilities of the MultiDomain-YOLOv8 model are expected to further expand its applications in various agricultural domains, contributing to the development of smart agriculture.
5. Conclusions
This study proposes the MultiDomain YOLOv8 model for the instance segmentation and classification of Sichuan pepper, along with a method for selecting high-quality Sichuan pepper. Extensive experiments demonstrate the superiority of the proposed algorithm. The study introduces the multi-scale frequency domain feature fusion module (MSF3M) and the multi-scale dual-domain feature fusion module (MS-DFFM), focusing on multi-domain and multi-scale feature extraction to enhance network performance. Additionally, modules such as Adown, C2f-SE, Fourier up-sampling, and RepNCSPELAN4 are incorporated to address issues related to high computational complexity and sampling inaccuracies in certain modules, further accelerating network inference and improving accuracy. The scientific validity and rationality of introducing these modules are confirmed through extensive experimentation.
Compared to the YOLOv8s-seg baseline network, the proposed MultiDomain YOLOv8-seg network shows an increase of 2.8% in accuracy and 1.3% in , with a reduction of 7.35 MB in parameters and GFLOPs of only 20.9. The MultiDomain YOLOv8-cls network achieves an accuracy of 98.34%, an FPS of 633 frames, and a model size of only 0.648 MB, outperforming the compared state-of-the-art models in various aspects. The proposed model makes significant advancements in reducing model size and computational requirements, making it more suitable for real-time applications. These improvements are expected to benefit the Sichuan pepper industry by enabling more precise and faster sorting operations, further supporting the development of agricultural sorting robots for Sichuan pepper.
In addressing the computational resource constraints in smart agriculture scenarios due to cost issues, the proposed network model, with its low parameter count and fast computation speed, is more advantageous for the design and deployment of sorting robots and network models for Sichuan pepper. Future work will focus on the practical application of the proposed algorithm in sorting robots, with ongoing improvements to address any arising challenges, ultimately achieving more accurate and rapid sorting and providing a reference solution for sorting high-quality Sichuan pepper.