Innovative Research on Intelligent Recognition of Winter Jujube Defects by Applying Convolutional Neural Networks

Abstract

The current sorting process for winter jujubes relies heavily on manual labor, lacks uniform sorting standards, and is inefficient. Furthermore, existing devices have simple structures and can only be sorted based on size. This paper introduces a method for detecting surface defects on winter jujubes using convolutional neural networks (CNNs). According to the current situation in the winter jujube industry in Zhanhua District, Binzhou City, Shandong Province, China, we collected winter jujubes with different surface qualities in Zhanhua District; produced a winter jujube dataset containing 2000 winter jujube images; improved it based on the traditional AlexNet model; selected a total of four classical convolutional neural networks, AlexNet, VGG-16, Inception-V3, and ResNet-34, to conduct different learning rate comparison training experiments; and then took the accuracy rate, loss value, and F1-score of the validation set as evaluation indexes while analyzing and discussing the training results of each model. The experimental results show that the improved AlexNet model had the highest accuracy in the binary classification case, with an accuracy of 98% on the validation set; the accuracy of the Inception V3 model reached 97%. In the detailed classification case, the accuracy of the Inception V3 model was 95%. Different models have different performances and different hardware requirements, and different models can be used to build the system according to different needs. This study can provide a theoretical basis and technical reference for researching and developing winter jujube detection devices.

1. Introduction

China is a country with a big fruit cultivation industry; since the 1980s, the cultivation scale and output of the Chinese fruit industry have been rising yearly, gradually surpassing those of European and American countries [1]. The varieties of winter jujubes are diversified; there are more than 700 kinds of winter jujubes in the records of China alone, as the winter jujube is an edible fruit unique to China, and its cultivation has a thousand years of history. In recent years, the winter jujube, with its thin skin, delicate meat, rich nutritional content, etc. [2], has gradually been popularized throughout the world. In 2023, China’s winter jujube production was approximately 3 million tons, with exports totaling 30,000 tons. Currently, picking operations of winter jujubes rely on multiple batches of manual picking and then their transport to a centralized manual sorting location. However, this kind of picking method results in the fruits being easily damaged, fruits with different maturities, both good and bad winter jujubes being mixed together, and so on, resulting in fruits that cannot undergo storage and long-distance transportation; additionally, it increases the winter jujube sorting difficulty, has serious impacts on economic benefits, and so on [3]. The best method for processing winter jujubes is to pick and store them for no more than 12 h to ensure that they reach the crispy ripening stage. The stored winter jujubes should be free from mechanical injuries, diseases, cracks, and other defects. [4]. Therefore, processing after picking is critical to enhancing the value of winter jujubes, but there is also a current urgent need to solve the technical problems. The grading of winter jujubes usually includes the following indexes—size, shape, maturity, and surface quality—of which the surface quality of winter jujubes is the most important [5].

At present, in China Zhanhua, Dali, and other winter jujube planting areas, winter jujube sorting work is mainly completed manually; sorting work occupies the labor force, accounting for 40% of the whole winter jujube industry chain [6]. With artificial sorting, there are many problems: due to the small size of winter jujubes, the artificial sorting accuracy and efficiency are low [7], and human subjective factors also lead to a winter jujube sorting standard that is not uniform, which seriously impacts the winter jujube grower’s income [8].

The grading and quality testing of fruits are important steps before they are stored or entered into the market, and consumers like products with good quality and good appearance. The testing and grading work of winter jujubes, on the one hand, can enhance the market competitiveness of winter jujubes and create a good reputation; on the other hand, eliminating unqualified winter jujubes is conducive to the long-distance transportation and storage of winter jujubes [9]. The winter jujube shelf life is short, and the existence of defective winter jujubes will cause them to quickly deteriorate and rot, especially if the maturity is too high and there are mechanical injuries to the winter jujubes, which will accelerate a winter jujube quality decline in the same batch [10]. The skin of winter jujubes is brittle and thin; just pressing hard on it can destroy the internal structure and form defects. How to realize the detection of the surface quality of winter jujubes and realize the non-destructive sorting of winter jujubes is thus the current problem.

Convolutional neural network (CNN) technology is widely used in various industries for the real-time detection and automated processing of objects. In the field of fruit products, a large amount of CNN-based sorting equipment that can realize more intelligent, efficient, and accurate fruit quality detection has also emerged [11]. This study, based on the current problems in winter jujube sorting in China and the analysis of existing agricultural product detection technologies, uses convolutional neural network technology and refers to the “National Standard of the People’s Republic of China—Winter Jujubes” to classify winter jujubes with different surface defects. A winter jujube surface defect recognition and detection model is constructed. This study can reduce production costs, improve the efficiency and accuracy of winter jujube sorting, and promote the industrial development of winter jujubes.

Currently, researchers both domestically and internationally have conducted extensive research on agricultural detection. This includes the use of convolutional neural networks for managing agricultural planting, such as detecting field weeds [12], detecting crop pests and diseases [13], and monitoring soil conditions. Additionally, research has been performed on detecting various agricultural products, including spherical fruits and vegetables, like apples [14], oranges [15], and tomatoes [16], as well as smaller fruits and vegetables, like grapes [17] and goji berries [18]. Furthermore, detection methods have been explored for irregularly shaped fruits and vegetables, such as potatoes [19] and strawberries [20].

From the aspect of agricultural production detection, this research mainly focuses on defect extraction and defect classification, and the application of machine vision and convolutional neural network technology is the common feature of these research methods. Machine vision technology can be used as the eyes of the computer to perceive the defects, the texture, the color, the type, and other information on the surface of the fruit, which is information that traditional spectral technology cannot provide; a convolutional neural network is used as the brain to screen and differentiate this information, and for the computer, different images will bring different information. Thanks to the high precision, high efficiency, lack of contact, and other advantages of this approach, the use of image processing to realize agricultural fruit and vegetable crop detection has become mainstream [21].

In general, a neural network simulates the process of information processing in a human brain, and a computer program recognizes the input information [22]. The earliest neural network model was the MP model, proposed by McCulloch and Pitts in 1943, and after continuous evolution, Rumelhart and Hinton et al. proposed the backpropagation network. In the 1990s, the BP neural network [23] and the theory of visual image recognition were further developed, with models such as support vector machine (SVM) [24] appearing one after another. However, problems such as local optimization, overfitting, and gradient diffusion could not be solved, so development slowed down. In 2006, Geoffery Hinton et al. thought that the network could be trained layer by layer to improve the feature learning ability, and this perspective in the field of artificial intelligence has caused a significant impact. Since then, scholars in many fields have proposed a variety of network models, and some scholars have attempted to combine neural networks and machine vision technology in the field of agricultural product inspection [13].

KC et al. proposed a separable convolutional architecture for plant pest and disease detection with a success rate of 98.65% [25]. Pattnaik et al. proposed a migration learning-based framework for tomato plant pest classification, achieving an 88.83% classification accuracy on the DenseNet169 model [26]. Al-Saif et al. devised a technique to distinguish between various varieties of Indian date fruits by using the color and morphological features of individual fruits and training an artificial neural network classifier, which achieved an accuracy of 97.56% [27]. Osako et al. developed a varietal classification system for lychee fruits using a pre-trained VGG16 model, which had an accuracy of 98.33% [28]. Singh et al. used histogram equalization to enhance the image and then applied the k nearest neighbor (KNN) classifier to identify two types of apple leaf diseases. The experimental results showed that the classification accuracy was 96.41% [29].

Although convolutional neural networks have made some progress in the detection and recognition of agricultural products, there are still many challenges to overcome, including a limited number of categories and low accuracy. In this study, we independently constructed a dataset of winter jujubes and divided them into six categories based on their different appearance qualities. We used a convolutional neural network to achieve accurate recognition of winter jujube images; improved the AlexNet; and conducted comparison experiments with VGG16, ResNet34, and InceptionV3, aiming to obtain a model with a more accurate recognition rate for winter jujubes. After accurately classifying the winter jujube dataset and improving the model, our model has gained the ability to accurately recognize and classify defects in winter jujubes.

2. Materials and Methods

2.1. Image Acquisition System Components and Development Platform Support

Image acquisition is the basis for recognizing the surface defects of winter jujubes, and convolutional neural networks are capable of learning key information from images. Therefore, constructing a reasonable dataset of winter jujubes is crucial to ensuring the accuracy of surface defect recognition. The main purpose of image preprocessing is to improve image quality, enhance the relevance of real information, eliminate interference from irrelevant information in the image, and thereby improve the reliability of the image feature law and the accuracy of model training.

The image acquisition device is shown in Figure 1, and its composition mainly consisted of a cardboard box, a fill light, a camera, and a computer. Since the image acquisition work was completed in the field, a cardboard box, which is easy to set up, was used as a dark box to avoid the influence of natural light intensity on image acquisition. The camera, the core component of the image acquisition, was installed on the top surface of the cardboard box to photograph the side of the winter jujube from top to bottom; a Samsung MV900F camera was used here, with an effective pixel count of 16 million. Inside the cardboard box, lighting equipment for supplemental light was also installed; a symmetrical light source can effectively avoid producing shadows during the winter jujube image acquisition process. Additionally, the fill light was an adjustable LED cool light purchased online for use as a source of illumination. After adopting the above method, although the problem of reflections on the surface of the winter jujube was largely solved, the effect of light on the surface of winter jujube was still inevitable. This is because it is not like other fruits, such as peaches, which have a rough surface and only slight reflections; the surface of a winter jujube is smooth. Therefore, we performed a data augmentation operation on the dataset to minimize the effect of reflections on the experimental results.

The development platform used for the experiment consisted of a hardware platform and a software platform. The hardware platform was a desktop computer with an Intel(R) i9-9900KF CPU, two NVIDIA RTX 2080 SUPER GPUs, 16GB RAM, and Windows 10 Professional. The software and tools required for the software platform were as follows: for image processing, the software included JetBrains PyCharm Community Edition 2019.2.3 ×64, Anaconda; the programming language was Python 3.7; and the programming libraries used included OpenCV-Python, NumPy, and Matplotlib. For convolutional neural network model training, the GPU version of TensorFlow 2.3.0 deep learning framework was used; the programming language remained Python 3.7; and the programming editor was Jupyter Notebook 6.3.0.

2.2. Research Target

The research object was the winter jujube produced in Zhanhua, Binzhou, Shandong Province, and the winter jujubes were collected from neighboring orchard areas in Xiawa Town and Dagao Town. To ensure that the data source was natural and representative, reflecting a real, natural ripening situation of winter jujubes, photography mainly focused on the fruit stalk, side, and defects of the winter jujubes.

We referred to the National Standard of the People’s Republic of China—Winter Jujubes GB/T32714-2016 [30]. The winter jujubes were divided into several categories based on differences in their surface quality, such as good, diseased, cracked, and bruised. Additionally, due to the large difference in shape and appearance of the high-quality winter jujubes, malformed fruits were generally sorted out; furthermore, fully ripe winter jujubes become soft over time, making their storage time shorter than semi-ripe fruits, so overripe winter jujubes were also sorted out.

Since some of the images contained multiple winter jujubes at the time of shooting, they needed to be cropped and uniformly scaled. Here, OpenCV tools were used for processing, and the final cut resulted in images containing only one winter jujube each with a size of 224 × 224 pixels. There were 1000 high-quality winter jujube images and 1000 poor-quality winter jujube images, totaling 2000 pairs of images in the initial dataset. Some of the high-quality images of winter jujubes are shown in Figure 2, and the number of images collected is shown in

Table 1. Types and quantity of images in datasets.

Image Type		Number of Images	Total
good		1000	1000
bad	overripe	187	1000
	cracked	174
	malformed	131
	bruised	192
	diseased	180
	blemished	136

.

2.3. Image Preprocessing

This part applies noise to the image before the enhancement operation. Image noise is typically generated by the camera during the image acquisition process or by the computer during processing and storage. The causes are likely changes in the environment where the image is acquired and interference from electronic devices. Noise usually exists in the form of isolated pixel dots or blocks of pixels within the image [31,32]. Here, noise was first imposed on the winter jujube RGB image [33], and then, filtering methods were used for processing and comparing the effects to decide which method to follow up with for image filtering.

After applying salt-and-pepper noise to the image, it was processed using four filtering methods, and the results are shown in Figure 3. It can be seen that the presence of noise greatly impacts image quality. Since noise is unavoidable and unpredictable, it can only be mitigated using other methods. Due to the overall filtering effect, a filtering method can reduce the impact of noise to some extent, but the relative quality of the original image will also be diminished, causing the image to appear fuzzy and hazy [34]. The filtering method with the best effect and the result closest to the original image was median filtering [35]. Bilateral filtering, on the other hand, had the worst effect, and many specks of salt-and-pepper noise could still be seen in the image.

The effects of each filtering method after applying Gaussian noise to the image are shown in Figure 4. From the figure, it can be seen that the effect of Gaussian noise on the image is much more severe than that of salt-and-pepper noise, which almost changes the color and contrast of some parts of the image. Compared with the original image, it can be seen that the image appears much more grey under the influence of Gaussian noise. Among the other images, the effects of each filtering method are worse, among which median filtering has the effect closest to that of the original image. The processing of Gaussian noise by the other three filtering methods was much less effective than the processing of salt-and-pepper noise. In summary, to reduce the effect of noise in the image, the first step should be to preprocess the winter jujube dataset using median filtering.

2.4. Dataset Preparation

The number of samples is directly related to the generalization ability and feature extraction ability of the convolutional neural network. A large number of training samples is the basis of neural network model training, and data augmentation can not only effectively increase the number of training samples [20] but also improve the accuracy and robustness [36] of the model training [37].

Two training programs were implemented: One involved classifying winter jujubes into two categories, with non-defective winter jujubes in one category and all defective winter jujubes in another. The second program involved a more detailed classification of winter jujubes, where each type of defective winter jujube was counted as a separate category, allowing for a screening of defective types. Program 1 enables quick model training by classifying only good and bad quality, with clear classification features that make the model easy to deploy for rapid detection or an assembly line operation. Program 2 provides the advantage of generating statistics on the output of winter jujubes for the current year, which can help date farmers manage their production more effectively in the following year.

2.4.1. Data Augmentation for Dichotomization of Winter Jujube Quality

Since the total number of images of defective winter jujubes was the same as the number of images of non-defective winter jujubes, the data augmentation method was adopted to expand the images of both defective and non-defective winter jujubes to 2000 images each. The operation methods included rotating the image by 30° clockwise and counterclockwise, mirroring in the vertical and horizontal directions, zooming in and out within the range of 90~110% of the original image size, and moving in the horizontal and vertical directions with a displacement of 20% [38]. One or two of the above image augmented methods were randomly selected, and the Python’s random function was used to randomly select the transformation values. The augmented data were normalized to produce 4000 input images of 224 × 224 pixels as the original dataset, and the dataset was partitioned according to the ratio of training set–test set–validation set = 7:2:1. Some of the expanded images are shown in Figure 5. The number of expanded data is shown in

Table 2. The number of winter jujubes after binary classification.

Image Type	Number of Images	Total
good	2000	4000
bad	2000

.

2.4.2. Data Augmentation for Detailed Classification of Data Quality

The collected defects of winter jujubes were classified according to GB/T32714-2016 and the national group standard information platform. The rust spots on winter jujube peel were due to blockage of the stomata on the fruit’s surface or necrosis of the surface cells. Therefore, blemished fruits were categorized as diseased fruits. Cracked and mechanically injured fruits were categorized as bruised fruits. Considering the varying frequency of each type of defective winter jujube, some types were captured in fewer images. Using the same expansion method as the second classification, the various images of defective winter jujubes were expanded to 400. Additionally, 400 images of non-defective winter jujubes were selected as a comparative group. The data were divided in the ratio of training set–test set–validation set = 8:1:1. The expanded data are shown in

Table 3. The number of winter jujubes after detailed classification.

Image Type	Number of Images	Total
good	400	2400
overripe	400
infested	400
malformed	400
bruised	400
diseased	400

.

2.5. Model Improvement and Evaluation Indicators

2.5.1. Model Improvement

The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) is one of the most sought-after and authoritative academic competitions in the field of machine vision in recent years, representing the highest level in the field of imaging. The AlexNet network was used in the ILSVRC classification competition with a subset of ImageNet that included tens of thousands of images and one thousand classifications. Since the purpose of this paper is to accurately distinguish whether there are defects on the surface of winter jujube, the structure of the original large-size convolutional kernel was modified based on the AlexNet model. This modification involved using multi-layer small-size convolutional kernels computed in series [39]; applying the BN (batch normalization) operation to the feature maps after the pooling layer; and inserting a Dropout mechanism after a specific BN layer to reduce overfitting. Additionally, the number of channels was reduced to improve training speed, among other improvements. The specific steps to improve AlexNet were as follows:

(1)

In order to ensure the scientific validity and effectiveness of the experimental results, to maintain the consistency of the data distribution between the training set and the test set, and to avoid the introduction of additional bias during the data partitioning process, the same preprocessing method was applied only to the winter jujube dataset. Images of the same size were used as input to the network. The input was an RGB true color image with an input layer size of 224 × 224 × 3, containing three color channels.

(2)

The characteristics of winter jujube fruits are relatively simple: they are nearly spherical, they have a simple color composition, their disease defects are obvious, and their ripeness is easy to detect ripeness, among other features. A model that is too shallow will be poorly trained, and one that is too deep is prone to overfitting, so the selection of the model and the appropriate adjustment of its depth are necessary. The model uses three 3 × 3 convolutional kernels in series, with the first convolutional kernel having a stride of 2, and the last two convolutional kernels having a stride of 1. The advantage is that the input and output sensory field sizes are guaranteed to be the same, which is the same size as the output feature map. The addition of two activation functions provides the model with a great nonlinear fitting ability for extracting the features of winter jujube. The output feature map size is shown in Equation (1).

$$N={\displaystyle \frac{W-F+2p}{S}}+1$$

(1)

where N is the output feature map size, W is the input feature map size, F is the convolution kernel size, p is the number of padding pixels, and S is the convolution kernel stride. A convolution kernel of size 11 × 11 outputs a feature map size of 1 × 1 after one convolution computation, whereas a 3 × 3 convolution kernel first convolution computation yields a feature map size of 5 × 5, its second convolution computation yields a feature map size of 3 × 3, and its third yields a size of 1 × 1. Similarly, a 5 × 5 convolution kernel is replaced by two 3 × 3 convolution kernels with stride 1.

(3)

In terms of defects, the disease defects of winter jujube are more distinguishable compared to normal fruit, but the feature extraction effect for inconspicuous features such as rust spots and knock marks is greatly reduced. Therefore, a BN layer needs to be added before each pooling layer to normalize the same latitude features of a batch of samples, forcibly turning the data into a normal distribution, which subsequently accelerates the model learning convergence speed. The extracted convolutional features are then compressed and pooled to prepare for subsequent feature fusion.

(4)

The Dropout model is trained with Dropout added after the last BN layer and two fully connected layers, with the Dropout rate set to 0.3. The number of channels is 64, 64, 64, 128, 128, 256, 256, 192, 2048, and 2048, and at this time, the model has 74,356,568 parameters. The structure of the improved AlexNet model is shown in Figure 6 below, where Conv2D represents the convolutional layer, S1 stands for a stride of 1, BatchNomalize is the batch normalization, MaxPooling is the maximum pooling operation, FC is the fully connected layer, and Softmax is the classification function.

2.5.2. Model Structural Parameters

The improved model results need to be compared with other classical convolutional neural network models to select the optimal model for building the visual detection system. Therefore, in other aspects of the settings, as far as possible, the same setting should be ensured for each convolutional neural network model. The model training epoch is set to 50, the training batch size is set to 32, and so on.

Adam [40] is an adaptive learning rate algorithm. Its advantage is that it can update the position based on the gradient, weighted according to the size of the gradient, to eliminate fluctuations in the gradient descent and then ensure that the gradient of each dimension is maintained at a level close to the optimal solution. In addition, Adam is used to optimize the algorithm in most cases, thanks to its incorporation of the first-order momentum of the SGD-M algorithm, the second-order momentum of the AdaGrad algorithm, and the RMSProp adaptive learning rate. This makes Adam a masterpiece of SGD. Moreover, Adam is not sensitive to the initial learning rate setup and can be optimized to a better parameter over a wide range of intervals.

The setting of the learning rate is related to the model structure. A suitable learning rate can help the model escape from the local optimum of these convex functions and reach the global solution of the model. The ReLu function is chosen as the activation function, and cross-entropy is chosen for the model’s loss function.

2.5.3. Evaluation Indicator

In order to verify the network model for winter jujube surface defect detection, this paper chooses the confusion matrix as the index to visualize the recognition of each category and calculates the accuracy, recall, and F1-score from them [41]. The structure is demonstrated using the confusion matrix for the three classifications, as shown in

Table 4. Composition of the confusion matrix.

	Forecast: A	Forecast: B	Forecast: C
True Classification A	TP	FN1	FN2
True Classification B	FP1	TN1	FN3
True Classification C	FP2	FN4	TN2

. It illustrates how the confusion matrix is calculated.

Considering the surface quality of winter jujubes, it is set that winter jujubes with good surface quality are positive examples and other winter jujubes with defective surfaces are negative examples. Here, actual classification A is a positive example, and B and C are negative examples. The prediction results for different kinds are distinguished by labeling. Therefore, in the table, we have the following definitions:

TP—true cases, the number of actual positive cases that were predicted to be positive cases;

FN—false negative examples, the number of actual positive examples that were predicted to be negative examples;

FP—false positive examples, the number of actual positive examples that were predicted to be positive examples;

TN—true counterexamples, the number of actual counterexamples that were predicted to be counterexamples.

Accuracy is the percentage of all samples where all predictions are correct, and in general, the closer accuracy is to 1, the better, as shown in Equation (2).

$$Accuracy={\displaystyle \frac{TP+T{N}_1+T{N}_2}{TP+T{N}_1+T{N}_2+F{P}_1+F{P}_2+F{N}_1+F{N}_2+F{N}_3+F{N}_4}}\times 100\%$$

(2)

The precision rate, also known as the positive predictive value, indicates the proportion of true positive categories among all the samples predicted as positive and is calculated as shown in Equation (3), where Precision(A) is the precision for classification A, Precision(B) is the precision for classification B, and Precision(C) is the precision for classification C. The closer the precision rate is to 1 for each classification, the better. The calculation is shown in Equation (3):

$$\left\{\begin{array}{l}Precision(\mathrm{A})={\displaystyle \frac{TP}{TP+F{P}_1+F{P}_2}}\times 100\%\\ Precision(\mathrm{B})={\displaystyle \frac{T{N}_1}{T{N}_1+F{N}_1+F{N}_4}}\times 100\%\\ Precision(\mathrm{C})={\displaystyle \frac{T{N}_2}{T{N}_2+F{N}_2+F{N}_3}}\times 100\%\end{array}\right.$$

(3)

Recall represents the proportion of all positive samples that are predicted correctly. The closer the recall for each classification is to 1, the better. Recall(A) represents the recall for classification A, Recall(B) represents the recall for classification B, and Recall(C) represents the recall for classification C. The recall for each classification is calculated as follows. The calculation is shown in Equation (4):

$$\left\{\begin{array}{l}Recall(\mathrm{A})={\displaystyle \frac{TP}{TP+F{N}_1+F{N}_2}}\times 100\%\\ Recall(\mathrm{B})={\displaystyle \frac{T{N}_1}{F{P}_1+T{N}_1+T{N}_3}}\times 100\%\\ Recall(\mathrm{C})={\displaystyle \frac{T{N}_2}{T{N}_2+F{P}_2+F{P}_4}}\times 100\%\end{array}\right.$$

(4)

F1-score is a comprehensive metric that balances precision and recall, and its value tends to be on the lower side. The simplified formula varies for different classification quantities, and the original calculation is shown in Equation (5):

$$\mathrm{F}1={\displaystyle \frac{2}{\frac{1}{Precision}+\frac{1}{Recall}}}$$

(5)

In addition, in the neural network model used to construct the visual system, regardless of whether the surface defects of winter jujube are classified in two or detailed classification, the most important point lies in the guarantee of defect-free fruits, which reduces the misdetection of defect-free fruits and ensures that other classified fruits will not be detected as defect-free fruits. Therefore, the model primarily uses the F1-score of defect-free fruits as a criterion for judgment, and the closer the F1-score is to 1, the more accurate the model’s detection of defect-free fruits becomes.

3. Results and Analysis

3.1. Training Results for Binary Classification

Although the Adam optimization algorithm was used, each model achieved different results at different magnitudes of learning rate. Therefore, under the same conditions of the training set, loss function, optimization method, etc., the surface defects of winter jujube were firstly subjected to binary classification training experiments, and the results are shown in

Table A1. Training results of each model in the binary classification case.

Model	lr	Assessment Set Accuracy/%	Assessed Set Loss Value	F1-Score	The Optimal Number of Training Epochs
AlexNet	10−4	96.49	0.1629	0.9188	40
	10−5	95.48	0.2046	0.9558	41
	10−6	97.24	0.1627	0.9747	32
Improved AlexNet	10−4	97.99	0.1301	0.9796	41
	10−5	99.21	0.0443	0.9924	29
	10−6	98.62	0.0723	0.9476	26
VGG 16	10−8	93.98	0.1884	0.9351	36
	10−9	96.49	0.1895	0.9487	30
	10−10	93.48	0.2833	0.8877	36
ResNet 34	10−3	97.24	0.1719	0.9697	37
	10−4	96.24	0.1763	0.9624	30
	10−5	91.72	0.3910	0.9173	47
Inception V3	10−3	97.75	0.1150	0.9698	43
	10−4	98.50	0.0733	0.9823	16
	10−5	94.23	0.4180	0.9409	17

in Appendix A. As can be seen from the table, even with the use of the Adam optimization algorithm’s adaptive learning, the effect that can be achieved by the model selection of different learning rates varied, and the learning rates at which each model achieved the best results were also different. In addition, the size of the learning rate was related to the step size of the model’s gradient descent. Choosing a learning rate that is too small can cause the model to fall into the local optimal solution and miss a better solution, such as VGG 16 with a learning rate of 10⁻¹⁰, ResNet 34 with a learning rate of 10⁻⁵, and Inception V3 with a learning rate of 10⁻⁵. The choice of learning rate can cause the model to oscillate around the optimal solution, such as AlexNet when the learning rate is 10⁻⁴, in a relatively simple structure with relatively few local optimal solutions; the set learning rate can make it jump out of one local optimum and into another, but it cannot make it converge stably. Therefore, the model is limited in its ability to reach the optimum on the evaluation set instantaneously. Then, for example, for a learning rate of 10⁻⁴ for the improved AlexNet, although the network depth has increased, the main structure is still a series one-way conduction computation, which is best evidenced by the lower accuracy rate at this point. The training curves of each model with the highest accuracy on the evaluation set are taken to show the training curves of each model with the optimal accuracy on the evaluation set in Figure 7.

From the above curves, it can be seen that the improved AlexNet model and the Inception V3 model have the best training effect, and there is not much difference between their performance on the evaluation set. In the early stage of training, the improved AlexNet converges faster, and the loss value has already converged to a smaller value in the 9th round; the advantage of the Inception V3 model is that it is more stable and fluctuates less after convergence, and the evaluation accuracy of the model is even refined at the end. In contrast, the training results of the other models are much more convoluted: the AlexNet model frequently fluctuates on the evaluation set and does not reach stable convergence, with the accuracy reaching a maximum of 97.47% only for a moment, and there are multiple peaks of loss values exceeding 1; the VGG 16 model exhibits constant fluctuations and maintains around 90%, which, if trained with the same magnitude of learning rate as the other models, would result in a much lower learning rate for the models. If we use the same learning rate as other models, it will lead to overfitting during training; the ResNet 34 model has an overfitting phenomenon during training, but thanks to the constant mapping transformation of its residual structure, its overfitting in the 22nd epoch returns to the convergence state quickly and gradually reaches a stable convergence.

The confusion matrix of each of the above models is shown in Figure 8, from which it can be seen that the improved AlexNet model, the Inception V3 model, and the ResNet 34 model have fewer erroneous detections. Overall, many models produce large errors in the detection of defect-free winter jujubes, and most of the models easily misclassify defect-free winter jujubes as defective. The reasons for this are analyzed as follows: on the one hand, the ripeness of winter jujubes is considered to be too large, and their freshness period is relatively short, so if they rot during transportation, it will affect the quality of other fruits; on the other hand, winter jujubes with defects such as rust spots are prone to detection errors because some rust spots are small and scattered, and the denoising of the image during image acquisition will also affect the rust spots, leading to the model’s poorer recognition of these rust spots. In addition, the optimal model is the improved AlexNet model, but it has a large number of parameters and is 850 MB in size, so when accuracy is a priority, you can choose the improved AlexNet model; the Inception V3 model has a small number of parameters and is only 260 MB in size, so if detection speed and the construction of a lightweight system are needed, you can choose the Inception V3 model.

3.2. Training Results for Detailed Classification

Due to the small amount of data in each classification case, only the improved AlexNet model and the Inception V3 model were trained and analyzed, with reference to the training results of the binary classification model. The results are shown in

Table A2. Training results of each model in the detailed classification case.

Model	lr	Assessed Set Accuracy/%	Assessed Set Loss Value	F1-Score	The Optimal Number of Training Epochs
Improved AlexNet	10−4	95.48	0.2046	0.8864	33
	10−5	96.49	0.1895	0.9000	39
	10−6	93.48	0.2833	0.8577	29
Inception V3	10−3	96.24	0.1763	0.9393	38
	10−4	97.24	0. 1627	0.9620	42
	10−5	94.73	0.1139	0.8941	34

in Appendix A.

The training curves of the improved AlexNet and Inception V3 models are shown in Figure 9, from which it can be seen that when the number of classifications increases, the accuracy of the Inception V3 model is significantly higher than that of the improved AlexNet model. The reason for this is that, with the same number of training epochs and the same optimization algorithm, the multi-scale convolutional computation of the Inception V3 model has a stronger feature learning ability than the multi-layer small convolutional kernel tandem computation of the improved AlexNet model. Especially in the case of multi-classification, various classification features are distinct, and the Inception V3 model can fit different scales of features to them and then form a shortcut-like computation for a certain feature within the model. Additionally, the Inception V3 model is structurally superior to the improved AlexNet model in terms of non-linear fitting ability, which makes its stronger feature learning ability reasonable.

The confusion matrix is shown in Figure 10. The overall performance of the improved AlexNet model is worse than that of the Inception V3 model, with 13 incorrect target distinctions, of which 4 are defective fruits classified as non-defective and 4 non-defective fruits classified as defective fruits. From the specific classification situation, it can be concluded that the two models perform extremely well in the classification problem of malformed and bruised fruits, which have relatively obvious features but perform relatively poorly in the classification of features that are similar, such as overripe fruits and non-defective fruits. In addition, there is a situation where some infested fruits’ images are recognized a diseased fruits. The performance of the detection equipment has less impact in this case, but if it is used for the current year’s winter jujube production data statistics, it has a greater impact. The presumed reason is the same as that above: part of the infested fruit image features and the diseased fruit image features are similar.

3.3. Model Validation

In order to verify the effectiveness of the trained model, winter jujubes were purchased from the Zhanhua area and Dali, respectively; 200 images of winter jujubes with various surface qualities were recollected; they were feed into the networks for verification tests of different classifications; and representative images were selected among them for analysis. The number of specific images for each type of surface quality is shown in

Table 5. Model validation dataset.

Image Type		Number of Images	Total
good		45	200
bad	overripe	36
	infested	22
	malformed	29
	bruised	37
	diseased	31

.

The validation program was written using the Pycharm Community Edition, with Python 3.7 as the programming language. The tools included TensorFlow GPU 2.3.0, os, OpenCV, matplotlib, numpy, and others.

For the surface defect binary classification model, the validation method of not filtering the image but only resizing it and then feeding it directly into the network was used. The resulting confusion matrix comparing the improved AlexNet model and the Inception V3 model is shown in Figure 11, and some of the detected images are shown in Figure 12.

Combining Figure 11 and Figure 12, it can be seen that the accuracy of the improved AlexNet model in the binary classification case is 98%, and the accuracy of the Inception V3 model is 97%. As analyzed above, the two models have poorer classification results for non-defective winter jujubes and overripe fruits, and weaker recognition effects for parts such as the skin of winter jujubes subjected to slight scratches and fruit tips. From the overall point of view, the validation results of the two models are high. The images used for validation were added to the background, and in this case, there were only a few images with detection errors, proving the feasibility of the two models for the binary classification of winter jujube surface defects.

For the detailed classification model of surface defects, only the Inception V3 model was used, and the image processing was the same as in the case of binary classification. The confusion matrix obtained is shown in Figure 13, and some of the detected images are shown in Figure 14. It can be seen that in the multi-classification case, although there are more errors in the identification of various defects, only two images produced errors in the prediction of defect-free fruits, and only one image was incorrectly predicted as a defect-free fruit. The images that produced errors in judgment were images with part of the fruit tip of the winter jujube, overripe fruits, small and scattered distribution of the surface color, and inconspicuous knock marks on bruised fruits. However, overall, the accuracy of the Inception V3 model reaches 95%, in which the F1-score value for non-defective winter jujube reaches 0.9663. The means that even if the use of the defective multi-classification model can be accomplished to extract defect-free winter jujubes, the ability to recognize defects in a variety of defective fruit defects is slightly weaker. Therefore, the model can be used for the construction of a visual inspection system.

4. Limitations of This Study

Although our study has made some progress on the classification task of winter jujubes, several limitations remain. Firstly, although the dataset we used contained multiple defective classes of winter jujubes, the number of samples in certain indistinguishable classes was relatively small, which may have affected the classification performance of the model. Secondly, although the neural network model we used performs well under specific conditions, its ability to generalize to new environments and different light conditions has not been fully validated. In addition, due to the limitation of computational resources, we were not able to explore larger-scale network structures, which may limit the potential for further performance improvement of the model. Finally, our evaluation metrics mainly focused on accuracy, while ignoring other performance metrics that may be more critical, such as robustness and real-time performance. Future work should focus on expanding the size and diversity of the dataset, optimizing the model architecture to improve generalization, and exploring a more comprehensive evaluation system. Limited computational resources during model training should be considered to reduce the model’s size, which also facilitates the deployment of the model on embedded devices at a later stage.

5. Conclusions

In order to explore models capable of detecting the surface quality of winter jujubes, this paper established two datasets for binary classification and detailed classifications of the surface quality of winter jujubes based on the grading standards and actual production demands. It adopted the simpler AlexNet model as the basis for the network deepening and structural improvement and conducted comparison experiments using a variety of classical convolutional neural network models. The simpler-structured AlexNet model was used as the foundation for deepening and improving the network structure, and various classical convolutional neural networks were employed for comparative experiments. The evaluation metrics included accuracy on the assessment set, loss value, F1-score, and confusion matrix to analyze the models in the two classification scenarios. It was verified that among the binary classification models, the improved AlexNet model achieved 98% accuracy on the validation data, while the Inception V3 model achieved 97% accuracy. For detailed classification, the Inception V3 model achieved an accuracy of 95%. The model validation confirmed the previous conjecture that the classification between defect-free and overripe winter jujubes is not distinct and also proves the feasibility of the model in both classification scenarios. Additionally, the improved AlexNet model and the Inception V3 model can be used to build a vision system. For the binary classification model, the improved AlexNet model offers higher accuracy, while the Inception V3 model requires less memory and has faster computation time. The Inception V3 model performs better when using the detailed classification model for defective winter jujubes, although the model’s detection performance decreases as the complexity of the classification increases.