Strawberry Defect Identification Using Deep Learning Infrared–Visible Image Fusion

Abstract

Feature detection of strawberry multi-type defects and the ripeness stage faces huge challenges because of color diversity and visual similarity. Images from hyperspectral near-infrared (NIR) information sources are also limited by their low spatial resolution. In this study, an accurate RGB image (with a spatial resolution of $2048\times 1536$ pixels) and NIR image (ranging from 700–1100 nm in wavelength, covering 146 bands, and with a spatial resolution of $696\times 700$ pixels) fusion method was proposed to improve the detection of defects and features in strawberries. This fusion method was based on a pretrained VGG-19 model. The high-frequency parts of original RGB and NIR image pairs were filtered and fed into the pretrained VGG-19 simultaneously. The high-frequency features were extracted and output into ReLU layers; the $l_1$-norm was used to fuse multiple feature maps into one feature map, and area pixel averaging was introduced to avoid the effect of extreme pixels. The high- and low-frequency parts of RGB and NIR were summed into one image according to the information weights at the end. In the validation section, the detection dataset included expanded 4000 RGB images and 4000 NIR images (training and testing set ratio was 4:1) from 240 strawberry samples labeled as mud contaminated, bruised, both defects, defect-free, ripe, half-ripe, and unripe. The detection neural network YOLOv3-tiny operated on RGB-only, NIR-only, and fused image input modes, achieving the highest mean average precision of 87.18% for the proposed method. Finally, the effects of different RGB and NIR weights on the detection results were also studied. This research demonstrated that the proposed fusion method can greatly improve the defect and feature detection of strawberry samples.

1. Introduction

Ripeness is a crucial factor for determining the best harvest time for fruits, which significantly influences the taste. During the ripening period, the skin of many fruits softens, and it is also easy to cause surface damage before and after picking, affecting the quality of sale. Judgment of ripeness and surface defects is complex and comprehensive. A reliable, rapid, and non-destructive measure will help to improve the work efficiency and reduce the artificial cost. With the development of computer technology, optical and imaging techniques have made these procedures tend towards simplicity.

Dhiman et al. [1] comprehensively summarized that RGB is the commonly used color space for some crop predictions and detection work currently. May et al. [2] used the RGB color model and fuzzy logic technique to assess the ripeness of oil palm fruit. Pardede et al. [3] studied the features extracted from RGB, HSV, HSL, and L*a*b* color space with the help of the support vector machine (SVM). Anraeni et al. [4] achieved strawberry ripeness detection by the RGB feature and k-Nearest Neighbor (k-NN) method, which performed well in the non-strawberry category. However, traditional fruit surface imaging methods only represent external information and are not sensitive to the internal information under the skin. Cubero et al. [5] and Costa et al. [6] summarized many commercial vision systems, but these measures only imitate human vision. Therefore, they only make judgments on the parts exposed to cameras. Moreover, because strawberries are herbaceous plants, the fruiting position is close to the land, which can easily cause mud contamination; the multiple colors of contaminants could make detection difficult. It has been proven that spectroscopy techniques can be used to evaluate the quality and ripeness of fruits or vegetables.

Hyperspectral imaging (HSI) is very useful for detecting bruises under the skin, and it can be used to evaluate changes in the fruit quality (e.g., soluble solids content (SSC), moisture content (MC), and firmness). Each pixel within a hyperspectral image contains abundant spectra, so a whole hyperspectral image is a three-dimensional cubic shape. And, more information can be involved in hyperspectral images than in visible images. Wei et al. [7] extracted hyperspectral features from wavelengths between 400 and 1000 nm to classify the ripeness of persimmon fruit. They made a different ripeness dataset with the help of a linear discriminant analysis (LDA), and reached a correct classification rate of 95.3%. Guo et al. [8] chose optimal wavelengths by principal component analysis (PCA) loading and fed features into the SVM to assess the ripeness situation of strawberries. Khodabakhshian et al. [9] and Benelli et al. [10] developed different regression models to detect fruit quality attributes and used classification algorithms to finally evaluate the grade of maturity.

Although HSI can provide richer spectral information, it still suffers from many challenges, including the need to select the appropriate spectral bands in advance, the limitation on results by the selected region of interest (ROI), the low spatial resolution, the slow imaging speed, the unintuitive imaging results, etc. In addition, similar spectral curves might sometimes be ambiguous. Elmasry et al. [11] found that the sharply changed spectral curve between 600 and 800 nm is optimal for evaluating the ripeness of strawberries, while Liu et al. [12] thought that the similaraties within the same spectral range could be used to evaluate fungal contamination of strawberries. Unfortunately, similar situations are often encountered in reality.

Furthermore, NIR imaging also has spectral information that is deficient in RGB images, and its imaging speed is faster than HSI as well. Luo et al. [13] proved that the best wavelengths for apple bruise detection are within the NIR range, and it is feasible to conduct detection by using the reflectance difference between the selected wavelengths. Wu et al. [14] used spectral preprocessing and the least squares support vector machine algorithm to detect tomato surface bruising within 600–1600 nm of the wavelengths. In a narrower spectral range, Wang et al. [15] reported that a 700–1000 nm spectral wavelength could be used to separate the chilling injured symptom severity of kiwifruits. Additionally, NIR wavelengths involved tasks including defect detection on chilling-injured nectarines [16], hailstorm damaged olives [17], and the sunburn [18] and bruise susceptibility of apples [19], which also achieved good results.

Unfortunately, current methods for NIR participation almost always acquire hyperspectra across the entire NIR range, from which some specific spectra in the ROI region are selected for analysis, and the the results are likewise limited by the selected area. This results in poor visibility of the final acquired image. Moreover, the presence of filters or spectroscopic prisms in NIR or hyperspectral cameras severely weakens the ability of CCDs to capture the light intensity, which is also a major factor causing spatial resolution degradation. Taken together, image fusion work is necessary to obtain a more visual and intuitive whole image.

Deep learning methods have gained popularity recently. Gao et al. [20] built a real-time HSI system with a pretrained AlexNet convolutional neural network (CNN) to evaluate the early ripe stage or ripe stage of strawberry. They used two wavelength ranges (528 and 715 nm) of the spectrum as the dataset in the laboratory and obtained an accuracy of 98.6% for the test dataset. Su et al. [21] developed one-dimensional and three-dimensional hyperspectral information [12] and fed it into 1D and 3D ResNet models to assess the ripeness and SSC of strawberries. They achieved an accuracy of over 84% but were limited by the insufficient capacity of the dataset. Gulzar [22] used a transfer learning technique to classify fruits, which not only improved the accuracy, but also effectively solved the challenge of having insufficient training data. But, the problem with spatial resolution degradation was still unresolved, which could negatively affect the performance of neural networks. More importantly, when faced with the co-existence of multiple conditions, such as bruises, diseases, maturity judgments, dirt masking, etc, it is particularly important to avoid the influence of extraneous conditions on the results in a reasonable way. Established experiments have rarely discussed the co-existence of multiple conditions; they are often designed with only one or two variables, which is unlikely in reality.

The main objective of this study is to fuse the RGB and NIR information into a single image with greater visualization and evaluate the strawberry ripeness and surface quality, focusing on spectral information and spatial information as well. In this experiment, we proposed a fusion method for RGB and NIR images. Firstly, the high-frequency information from the RGB image and NIR image was extracted using filters, and the pretrained VGG-19 network was utilized to process the high-frequency part. Next, the high-frequency information was extracted in advance from RuLU layers with different activation functions, and the image pixels with different features of high-frequency information were averaged and processed to fuse them into a single image. Finally, the high and low-frequency images were fused together. The resulting fused image obtained both spatially resolved information and spectral information, which improved the accuracy of subsequent detection processes.

2. Materials and Methods

2.1. Strawberry Samples

A total of 240 strawberry (Fragaria × ananassa Duch.) samples (‘Hongyan’) were harvested by professional farmers from a local fruit growing base in Shenzhen, Guangdong province, China. These strawberry samples were divided equally into three groups: ripe stage, half-ripe stage, and unripe stage. The ripeness of samples was judged according to the color of the skin [23,24,25]. After being washed, wiped, and numbered individually, the samples were stored in a thermostat where the relative humidity was set to 70%, and the temperature was set to 15–22 °C. Among each group, 35 to 50 strawberries were artificially contaminated with mud, bumped, and rubbed. The firmness of the samples differed depending on the ripeness stage. Therefore, the bruises were different after the same level of treatment. This also exactly reproduced the state of the real transport situation. These strawberries were stored in a refrigerator at 5 °C for 1–3 days, during which they were randomly selected and photographed. The strawberry samples showed different damage states, which enriched our dataset.

2.2. Infrared Image and RGB Image Acquisition

2.2.1. Infrared–RGB Imaging System

Images of the strawberry samples were acquired by the hyperspectral reflectance system shown in Figure 1. The core component of the system was a push-broom hyperspectral imager (GaiaField-V10E, Dualix Instruments, Chengdu, China), which contained a hyperspectral imaging part and an RGB imaging part. The former part covered the Vis/NIR wavelength ranging from 400 nm to 1100 nm with a resolution of 2.8 nm. The hyperspectral CCD contained $696\times 700$ pixels, and each spectral image contained up to 256 bands (wavelengths). The RGB imaging lens was right next to the hyperspectral imaging part; the compact design allowed the two images to be viewed without too much difference in the viewing angle. The RGB imaging CCD contained $2048\times 1536$ pixels. Above all, two 250 W tungsten halogen lamps (Projection Lamp Type 13162, Philips, Germany) were fixed at the top. The rotatable platform connected with a stepping motor allowed the imager to acquire information from multiple viewpoints.

2.2.2. Image Acquisition and Preprocessing

The imaging system could acquire RGB images and NIR (700–1100 nm) images at the same time. Due to the much longer wavelength of NIR, the resolution of the NIR images was dramatically reduced. The acquisition range of the hyperspectral camera was fixed at 700–1100 nm [26], and 146 bands (channels) were contained in each image cube. And, the raw RGB sample images are shown in Figure 2. These included different defects and ripeness stages. It should be noted that although the hyperspectral camera was used in this experiment to acquire images in the NIR range, in the following work, the spectral image cube was added up into a single channel gray scale map that was exactly equal to the spectral integration effect of the NIR camera. This was because a suitable NIR camera was not available in the laboratory, and secondly the small interval between the two lenses on the hyperspectral camera facilitated subsequent image registration.

The initial images should be corrected by the black reference image and white reference image under the unchanged dark room environment. The dark reference image was captured by covering the camera with a completely opaque lens cap and turning off the illumination; the white reference image was captured by shooting a standard whiteboard with the illumination on. The final corrected images were calculated by Equation (1):

$$I_c=\frac{I_{initial}-I_{dark}}{I_{white}-I_{dark}}$$

(1)

where $I_c$ is the image after correction, $I_{initial}$ is the raw image output by the imager, $I_{dark}$ is the dark reference image, and $I_{white}$ is the white reference image. The whole correction procedure was executed by the software attached to the imager.

Common NIR images have a single channel, and each image cube should be transformed into the same form. This procedure was based on Equation (2):

$$I_{fused}(x,y)=\sum _{i=1}^N\frac{I_c^i(x,y)}{N}$$

(2)

where $I_{fused}$ is the single-channel gray-scale image after spectral fusion, $(x,y)$ is the spatial coordinate within each image, N is all channels within the NIR range, which was set as 146, and i represents the i-th channel in each image cube.

After obtaining the three-channel RGB image and single-channel NIR gray-scale image, due to the different shooting views, registration processing should be performed between the NIR and RGB images. Cross-correlation method [27] was consulted to align and calibrate the pixels of two source images. According to the similarity between the RGB image and NIR image, each pixel was matched to avoid blurring and ghosting.

2.2.3. Dataset Preparation

The data included RGB images and the corresponding NIR images, which were captured from the same scene and registered. With each exposure, the rotatable platform rotated at a random angle. Each strawberry was photographed three to five times. The total set of samples included 1000 RGB images and 1000 NIR images from 240 samples. Before being divided into different sets, all image pairs were fused by the proposed method.

All fused images were divided into three groups according to the ripeness stages. As mentioned above, a certain number of samples were contaminated, bumped, and rubbed randomly within each group, so that each fused image contained specific features of relative samples. The fused images were manually labeled with the tool LabelImg (URL: https://github.com/tzutalin/labelImg, accessed on 1 October 2022), and 80% of the fused images were randomly selected to build the training set, and the remaining images were employed as the testing set. To overcome the issue of having insufficient data, the expansion procedure was operated before training the deep learning detection network, as follows: (1) flipping each fused image horizontally along the vertical direction; (2) scaling each fused image 0.8–1.5 times randomly; (3) panning each fused image in four directions (up, down, right, or left). By using the above three approaches, the size of the dataset was expanded from 1000 raw fused images to 4000 fused images. The training set and testing set were divided according to the ratio of 80% and 20%. In other words, after removing the invalid images, the augmented training set contained 3150 fused images, and the augmented testing set contained 783 fused images. The detailed information about the training set and testing set is demonstrated in

Table 1. Detailed labels for each ripeness stage of the strawberry training set.

Types	Ripe	Half-Ripe	Unripe
	1034	1124	992
Contamination	224	278	253
Bruises	231	234	214
Both defects	180	269	203
Defect-free	399	343	322

and

Table 2. Detailed labels for each ripeness stage of strawberry testing set.

Types	Ripe	Half-Ripe	Unripe
	255	283	245
Contamination	57	68	62
Bruises	56	56	52
Both defects	47	68	51
Defect-free	95	91	80

.

Labels, including the three ripeness stages, mud pollution, and mechanical damage, were manually placed on fused image samples, as shown in Figure 3. The background and ROI did not need to be predefined because the neural network only paid attention to the whole sample regions after training.

2.3. Proposed Framework

The convolutional neural network (CNN) is a very effective machine learning method. Because of its high adjustability, generality, and large amount of data, it learns through data, extracts data features, and thus identifies complex model patterns [28]. The creation of the CNN eliminated the need for researchers to solve difficult mathematical formulas and physical problems. VGG-19- [29] or VGG-based models have been proven to be commonly used and reliable network models for image analysis and recognition. They were developed by the Visual Geometry Group from the University of Oxford. The simple network structure can meet the needs of secondary development according to different tasks, so this is the main reason why VGG series networks are favored by researchers in the agriculture and biology domains [30]. Previous works validated VGG-19 for use in vegetable texture recognition [31] and the classification of fruit ripeness [32]. Moreover, in the domain of RBG research images, it is easier to acquire and annotate the image source. Mamat et al. [33] used YOLOv5 to annotate fruit varieties, which made data acquisition more convenient. But, the advantages of RGB-NIR fused images were not fully exploited. A pretrained VGG-19 based network is used to extract image features, fuse the RGB and NIR images, and classify the different groups of strawberry samples in the following sections.

2.3.1. Fusion of the Low-Frequency Part

The convolutional kernels in VGG-19 can be seen as different types of filters, and the pretrained model automatically adjusts the parameters of each convolutional kernel. In this experiment, the pretrained model of ImageNet [34] was implemented. The high-frequency (HF) part of the images contained much richer information than the low-frequency (LF) part; therefore, the method described in [35] was used to decompose each source image pair into LF and HF parts. The LF parts were fused directly by the weighted-averaging strategy; the fused LF part was calculated by Equation (3):

$$F_{LF}(x,y)={\alpha }_{rgb}{I}_{LF}^{rgb}(x,y)+{\alpha }_{nir}{I}_{LF}^{nir}(x,y)$$

(3)

where $(x,y)$ denote the corresponding positions of the image intensity of the LF RGB image $I_{LF}^{rgb}$ and the LF NIR image $I_{LF}^{nir}$. ${\alpha }_{rgb}$ and ${\alpha }_{nir}$ represent the weights of $I_{LF}^{rgb}$ and $I_{LF}^{nir}$, respectively. In the following experiment, changes in the two weight ratios had little effect on the final results, so ${\alpha }_{rgb}$ and ${\alpha }_{nir}$ were set as 0.5.

2.3.2. Texture Extraction of the High-Frequency Part

For the HF contents, $I_{HF}^{rgb}$ and $I_{HF}^{nir}$, the proposed VGG-19-based framework was used to extract texture features. Two forms of images were extracted in the same way from the network.

In the pretrained VGG-19 network, parameters were adjusted to the optimal situation and fixed, because the channels of the VGG-19 were $M=64\times 2^{i-1}$, $i\in \{1,2,3,4\}$ with the changes in the i-th layer. The feature maps of the input images were extracted by convolutional kernels and were output at activation layers, denoted by $ReL{U}_1$, $ReL{U}_2$, $ReL{U}_3$, $ReL{U}_4$, and $ReL{U}_5$.

$$T_{rgb,nir}^i={ReLU}_i\left(I_{HF;rgb,nir}^i\right)$$

(4)

where $I_{HF;rgb,nir}^i$ is the high-frequency RGB image or the NIR image fed into the i-th activation layer, $ReL{U}_i(·)$ is the activation ReLU function processed within the i-th layer, and $T_{rgb,nir}^i$ is the image feature extracted and output by the i-th layer. $T_{rgb,nir}^i$ is a multidimensional vector, depending on the channels of its layer. The structure of the neural network and the output feature are shown in Figure 4.

The abundance of information can be measured by the pixel intensity within the images, and each input image corresponds to hundreds of feature maps. Using the $l_1$-norm [36], the feature maps corresponding to each input image were added up with Equation (5), and the information characteristics of each feature map were emphasized. This also took full advantage of the multiple convolutional kernels in neural networks when extracting features.

$${\widehat{T}}_{rgb,nir}^i=\sum _{m=1}^M\left|T_{rgb,nir}^{i,m}\right|$$

(5)

where $T_{rgb,nir}^{i,m}$ is the $l_1$-norm processed RGB or the NIR image output in the i-th layer, whose channel is $M=64\times 2^{i-1}$. Along the channel dimension, after adding up all the feature images, ${\widehat{T}}_{rgb,nir}^i$ was total output feature map in each activation function layer. To avoid the effect of extreme pixels on the fused image, area pixel averaging was introduced to make the algorithm robust with Equation (6).

$${\overline{T}}_{rgb,nir}^i=\frac{{\sum }_{\Delta x=-r}^r{\sum }_{\Delta y=-r}^r{\widehat{T}}_{rgb,nir}^i(x+\Delta x,y+\Delta y)}{{(2r+1)}^2}$$

(6)

where r is the area size. In this paper, $r=1$, so more texture information was preserved with guaranteed algorithm robustness.

2.3.3. Fusion of RGB-NIR Images

As the network became deeper, the size of the feature image output by the activation function became smaller because of the pooling operator. An up-sampling operator was introduced to restore the output feature images of each layer to the original images $I_{HF}^{rgb}$ and $I_{HF}^{nir}$.

In each output layer, up-sampled ${\overline{T}}_{rgb,nir}^i$ images were added up, and the output images of all five layers were added up as well with Equation (7). The fused HF part was obtained:

$$F_{HF}(x,y)={\sum }_{i=1}^N{\alpha }_{rgb}{\overline{T}}_{rgb}^{i\uparrow }+{\alpha }_{nir}{\overline{T}}_{nir}^{i\uparrow }$$

(7)

where ${\overline{T}}_{rgb}^{i\uparrow }$ and ${\overline{T}}_{nir}^{i\uparrow }$ are the up-sampled ${\overline{T}}_{rgb}^i$ and ${\overline{T}}_{nir}^i$, respectively, and $N=5$ is the number of activation layers. The extraction and processing flow of the HF image feature image are shown in Figure 5.

Since the fused LF part and fused HF part were obtained, the final fused RGB-NIR images was generated by Equation (8):

$$F(x,y)=F_{LF}(x,y)+F_{HF}(x,y)$$

(8)

2.4. Detection Evaluation

In order to test the performance of fused images in the detection process, we introduced evaluation metrics to measure the results. The detection of strawberry ripeness stages, defect situations, and mud contamination depending on different image inputs could be evaluated by the precision (P) and recall (R), which were calculated with Equations (9) and (10) [37]:

$$P=\frac{TP}{TP+FP}$$

(9)

$$R=\frac{TP}{TP+FN}$$

(10)

where TPs, FPs, FNs, and TNs represent true positives, false positives, false negatives, and true negatives, respectively. The relationships among the evaluation symbols, detection results, and labels are shown in

Table 3. Confusion matrix for the detection results.

Labels	Predicted	Evaluation Symbols
Positive	Positive	TP
Positive	Negative	FN
Negative	Positive	FP
Negative	Negative	TN

.

To further evaluate the results of the detection model, according to Rezatofighi et al. [38], the intersection over union (IoU) was used to define the overlapping area of the prediction box and ground truth box. In this paper, if the IoU was greater than 0.7, the prediction result was considered to be a TP. Otherwise, the prediction result was considered to be a FP. Based on the IoU, the P and R were calculated, and the average precision (AP) and mean average precision (mAP) were calculated to further assess the performance with different image inputs. Greater values of P, R, AP, and mAP indicate a better detection performance. The AP and mAP are calculated respectively as shown in Equations (11) and (12).

$$AP={\sum }_{n=1}^N\left(R_n-R_{n-1}\right)P_n$$

(11)

$$mAP=\frac{1}{K}{\sum }_{i=1}^{K}A{P}_i$$

(12)

where n is the index of the different confidence threshold points on the PR curve, $P_n$ is the corresponding precision value, and $R_n$ is the corresponding recall value. K is the total number of sample classes, and ${AP}_i$ indicates the average precision of the i-th class.

3. Results and Discussion

3.1. Evaluation of the Fused Images

One of the key points of this study was to propose an image fusion algorithm that facilitates the detection of fruit quality. Fused images must be evaluated as this directly affects the subsequent detection performance. Four quality metrics, the mutual information (MI, based on the information measure) [39], multi-scale structural similarity (MS-SSIM, based on the structural similarity measure) [40], standard deviation (SD, based on the image feature evaluation) [41], and modified fusion artifacts measure ($N_{abf}$, based on the noise evaluation) [42], were employed to assess the fusion performance.

In order to compare the fusion performance of the proposed method with that of the efficient algorithm, three typical fusion methods were selected. DenseFuse [43], FusionGAN [44], and U2Fusion [45] were introduced and compared with our proposed method. The compared methods are widely used in many fields [46,47,48], including object detection, recognition, and tracking. From the image data, 25 RGB and NIR image pairs and their fused images calculated using different fusion methods were randomly selected. These selected images have not been labeled yet, as only the fusion performance was verified in this section. The visible fusion results of different fusion methods are shown in Figure 6.

The average values of the four quality metrics for the proposed and comparative methods are shown in

Table 4. The averaged values of the four quality metrics of the strawberry fused images.

Methods	MI	MS-SSIM	SD	${\mathit{N}}_{\mathit{abf}}$
DenseFuse	12.95475	0.93145	65.12324	0.08453
FusionGAN	12.65822	0.77246	57.23540	0.06904
U2Fusion	13.55489	0.92492	64.01243	0.30145
Proposed	14.65096 *	0.93342 *	69.16351 *	0.05675 *

Note: The best results are denoted with *.

.

From Table 4, the proposed fusion method obtained the best values of MI, MS-SSIM, SD, and $N_{abf}$ compared to the other methods. A reliable fusion result was essential for subsequent detection.

One of the strengths of the proposed method is that the convolutional kernels of the pretrained network are automatically and apparently randomly fixed to the optimal values. This makes it more efficient and more flexible than the traditional manually defined filtering algorithms. First, with the more varied feature outputs, the final fused images contain more complete information. Therefore, the MI and MS-SSIM indexes were the largest for the compared methods. Second, multi-feature output layers and the area pixel averaging strategy reduced the influence of extreme pixels and there were few unobjective filters and little noise. Hence, the SD and $N_{abf}$ indexes performed well. The proposed fusion method obtained a good fusion performance, produced more content restoration, and had high structural integrity and lower noise. This was very convenient for the following detection and classification work.

3.2. Performance of Fused Image in Detection

3.2.1. Preparation of Detection Network and Dataset

The detection neural networks contain part of the functions of the classification neural networks. Detection networks are being used for the recognition of fruit and vegetable characteristics. In this field, the YOLO series of networks is very popular and is constantly being updated [49,50,51]. Additionally, Yang et al. [52] and Tian et al. [53] compared multiple types of YOLO networks with other detection networks in detail, reaching the conclusion that the YOLO series networks perform better in complex detection environments. YOLOv3-tiny [54] achieves a balance between speed and accuracy. It has the advantages of a smaller parameter volume, faster calculation speed, and high accuracy, which were validated in [52]. In this paper, YOLOv3-tiny was employed to train and test the fused images generated by the proposed method and was compared with the results obtained from the general image input. For the processing of parameter settings for training and testing, we referred to Yang et al. [52] and Shi et al. [54].

The training and testing sets of YOLOv3-tiny were divided as shown in Table 1 and Table 2. The hyperparameters are demonstrated in

Table 5. Key hyperparameters of the YOLOv3-tiny detection network.

Parameter	Value	Parameter	Value
Batch size	64	Optimizer	Adam
Epochs	100	Regularization	$L_2$ regularization
Learning rate	0.001	Input image size	$416\times 416$

, and the loss curves are shown in Figure 7.

3.2.2. Comparison of the Fused Image Input with the Traditional Input

Traditional input methods include only the inputting of RGB images or NIR images directly. The processes of labeling on RGB and NIR images are the same as those of fused images. The manually labeled anchor box coordinates and corresponding labels on the fused images could be directly transferred to the RGB and NIR images. The network parameter settings for the training and testing processes were the same for all three contrasting experiments. For the randomly selected valid testing set, 255 images/samples were in the fully ripe stage, 283 images were labeled as half-ripe, and 245 images were labeled as unripe. More detailed labels are shown in Table 2.

For the detection results based on the fused images,

Table 6. Detection results of the four quality situations and ripeness stages for strawberry samples in the testing set by YOLOv3-tiny based on fused images.

		R					HR					U					P (%)	R (%)
		Con	B	BD	DF		Con	B	BD	DF		Con	B	BD	DF
R	Con	47	3	2	0		1	1	0	0		2	1	0	0		90.38	82.46
	B	1	50	1	0		2	2	0	0		0	0	0	0		84.75	89.29
	BD	1	1	44	0		0	0	1	0		0	0	0	0		86.27	93.62
	DF	0	1	0	87		0	1	1	4		0	0	0	1		91.58	91.58
HR	Con	2	1	0	0		59	1	2	0		2	1	0	0		85.51	86.76
	B	0	2	1	0		2	46	2	0		1	1	1	0		77.97	82.14
	BD	1	1	2	0		2	2	56	0		0	1	3	0		86.15	82.35
	DF	0	0	0	6		0	1	0	79		1	0	0	4		85.87	86.81
U	Con	0	0	0	0		2	2	1	0		52	3	2	0		85.25	83.87
	B	0	0	0	0		0	1	0	3		1	39	2	6		79.59	75.00
	BD	0	0	0	0		1	2	2	0		2	3	41	0		83.67	80.39
	DF	0	0	1	2		0	0	0	6		0	0	0	71		86.59	88.75

shows the performance on the three ripeness stages and four quality situations. And, for record purposes, Con, B, BD, and DF were used to denote contamination, bruises, both defects, and defect-free; R, HR and U were used to denote the ripe stage, half-ripe stage, and unripe stage, respectively. In the ripe stage, the proposed method achieved values of 84.75–91.58% P and 82.46–93.62% R; in the half-ripe stage, the proposed method achieved values of 77.94–86.15% P and 82.14–86.81% R; and in the unripe stage, the proposed method achieved values of 79.59–86.59% P and 75.00–88.75% R. It is worth emphasizing that compared with the ripe stage, the P and R of bruises in the half-ripe and unripe stages significantly decreased, thus causing the occurrence of both defects to decrease as well. This is because, as the strawberry samples ripen, their skin becomes softer and is more vulnerable to damage [55]. Using the same treatments to bump and rub the samples, the bruises on the skin of the ripe strawberries would definitely be larger and more visible. Furthermore, the task of the discrimination of sample ripeness stages is more inclined to be a classification process, and the output of the YOLOv3-tiny is the ripeness level of the whole strawberry sample without the help of IoU to measure it. After calculating, the proposed method achieved values of 92.61% P and 93.33% R for the ripe stage, 88.42% P and 89.05% R for the half-ripe stage, and 92.12% P and 90.61% R for the unripe stage. Another reason for the low P-R values detected in the half-ripe stage group was that the standard between the ripe and half-ripe stages as well as the unripe and half-ripe stages was not specific and could cause some misjudgment.

Compared with the results for the fused images, the values of P and R in the RGB-only based group and NIR-only based group were also calculated in the same way. As shown in

Table 7. The values of P and R for four quality situations and ripeness stages in the strawberry sample testing set by YOLOv3-tiny based on the RGB-only and NIR-only methods.

		RGB-Only			NIR-Only
		P (%)	R (%)		P (%)	R (%)
R	Con	79.59	68.42		86.00	75.44
	B	76.36	75.00		83.93	83.93
	BD	70.91	82.98		78.43	85.11
	DF	88.17	86.32		90.22	87.37
	Ripe	91.67	90.59		89.56	87.45
HR	Con	78.13	73.53		80.30	77.94
	B	69.35	76.79		75.81	83.93
	BD	74.32	80.88		80.30	77.94
	DF	88.24	82.42		82.02	80.22
	Ripe	87.37	87.99		83.75	83.75
U	Con	80.33	79.03		82.26	82.26
	B	75.00	69.23		74.14	82.69
	BD	75.93	80.39		82.69	84.31
	DF	86.75	90.00		87.34	86.25
	Ripe	91.87	92.24		86.85	88.98

, the P and R values of the RGB-only based group were overall obviously lower than those of NIR-only based group and fused-image-based group. Especially, the detection performance on contaminated and bruised samples was the worst with weighted average P and R values of only 75.72% and 76.21%, respectively. According to the results, it was difficult to distinguish well between the contaminated and bruised samples based only on the RGB images. Different contaminant colors and bruises could become mixed up, making the detection networks ineffective. But, the results of ripeness detection were outstanding and were even comparable to the fused-image-based results. This was because of the accumulation of several anthocyanins that gave the ripe strawberries their red color [56] as they ripened, which could be easily reflected and detected by RGB images.

For the NIR-only-based group, the overall performance was between those of the RGB-only and fused-image-based groups. Compared with the proposed method, the R of the quality in the NIR-only-based group was smaller, which was caused by the misjudgment of the ripeness in each situation, which also decreased the P of other situations within each group. The NIR-only-based group achieved a good performance for the classification of contaminated and bruised strawberries; especially in the half-ripe and unripe stage, the stains (or bruises) on the harder skins were better suited for analysis by NIR images. The lower P values needed to be compensated by the excellent judgment performance of the RGB-only-based method for ripeness. This provided clear evidence of the need for the proposed method.

Given the evidence, it can be seen that bruises and contamination of the skin of samples was distinguished mainly by the reflectance of the target regions within the NIR wavelength range, and the classification ofthe ripeness stages was strongly associated with the RGB images. The traditional methods were used to divide the images into several ROIs and calculate the mean relative reflectance spectra [57] within each ROI. Each bruise or stain had different characteristics in different spectral ranges; however, in the visualized images, the mean relative reflectance could be completely represented by the gray-scale values in the NIR image [58]. These confirmed the feasibility of using the whole NIR image for classification. The detection results of six samples using the three methods are shown in Figure 8. This allows us to visualize the detection performance of the different methods.

The AP values of the four quality situations and ripeness stage detection are shown in Figure 9. The proposed method (mAP of 87.18%) has 5.42% and 3.61% higher performance levels in terms of the mAP metrics than the RGB-only-based method (mAP of 81.74%) and the NIR-only-based method (mAP of 83.57%), respectively. This is caused by the uneven performances of the two traditional methods. The proposed method properly combines the advantages of the two traditional ways and thus yields good results.

However, in the comparison experiments, the initial settings of the fusion weights for RGB and NIR were 0.5 and 0.5, but the optimal weighting ratio is not necessarily 1:1 in different practical situations and tasks. In the next experiments, the performance of the proposed method was studied under different RGB and NIR weighting ratios.

3.2.3. Comparison of Different RGB and NIR Weights

In the previous discussions and experiments, the fusion weight of both RGB and NIR was set as 0.5, i.e., the high- and low-frequency RGB and NIR had the same percentage in the final fused image. Although, when comparing the RGB-only- and NIR-only-based groups, the proposed method obtained excellent results in comparative experiments on our self-made dataset. It is apparently not practical for RGB and NIR to always have the same weights in multiple tasks. In order to study the specific performance of fused images with different weights of RGB and NIR parts, a new experiment was designed to research which detection tasks the different RGB/NIR weights are best suited to. The locations of the anchor boxes and corresponding labels were fixed, as in the previous experiment. The weight of RGB was used as the main reference object, and the weight was changed from 0.1 to 0.9 (the corresponding NIR weight changed from 0.9 to 0.1), with 0.05 being the minimum change scale.

According to Section 3.2.2, we initially learned that good detection results of quality situations and ripeness stages were not available at the same time in the RGB-only-based and NIR-only-based groups. This meant that the two detection tasks mainly relied on the information from the different sources of images. It was essential to measure the quality situation detection and ripeness stage detection separately to obtain the optimal weighting matches for different tasks. Based on this division method, the graph of results with the changes in the RGB/NIR weights is demonstrated in Figure 10. As the weights of NIR and RGB in the fused image varied, both the ripeness AP and quality AP exhibited significant changes. We observed that the ripeness AP reached its maximum value when the NIR weight was 0.35. Meanwhile, the quality AP achieved its highest value when the NIR weight ranged from 0.35 to 0.45. To show the results more briefly, the detection performance is shown in Figure 11 with a weight ratio of 0.1 as the minimum scale of variation.

Taking the RGB weights as the main reference, the scatter of both APs appears as maximal in the range of variables. The trends of the two tasks do not change monotonically. The detection of quality situations obtained the best performance, where the RGB weight was about 0.4, while the best detection of ripeness stages was obtained at around 0.7. This means that it is impossible to find a suitable weight for all detection tasks. The assignment of weights can only be chosen based on the main purpose of the task, so it is appropriate to separately discuss the tasks that the two information sources apply to. The quality AP decreased rapidly and obviously after achieving its maximum value at a weight of around 0.4; this was because the various colors of contamination and bruises confused the detection model in the RGB color space, and the NIR information was necessary for detection. But, before the quality AP reached its maximum value, the overall spatial resolution of the fused image was too low due to having too many components in the NIR band, which was also detrimental to the detection results [42,43,47]. This is an important reason why the mean relative reflectance of the ROI is mostly chosen as the object of study in NIR analyses. RGB images are much more sensitive to colors than NIR images, so when detecting the ripeness stage, especially when the ripeness could be judged well and accurately by red (ripe) and green or white (unripe) degrees, as the RGB weight increased, the overall ripeness AP was on an upward trend. The maximum value of ripeness AP appeared at a weight of around 0.7, and the AP did not fall back significantly as the weight increased; the small decrease that occurred might be due to ambiguous definitions and errors that occur when producing the sample set.

Taken together, this section of study supports the idea that the use of fixed weights of RGB and NIR in fused images for strawberry sample feature detection is not always the best option. In order to cope with multiple complex tasks, a system with the ability to choose weights flexibly is essential.

4. Conclusions

Strawberry is a kind of fruit with soft skin, which can be easily damaged or stained, thus affecting the commercial value. Many researchers have detected the ripeness and defects of strawberries based on RGB or NIR images. However, many limitations have often been encountered: It is difficult to detect defects under skin and contamination with similar visual colors in RGB images; NIR images can capture the difference in reflectance bands of defects and contamination, but are challenged by the severe lack of spatial resolution in the images. In order to combine the advantages of NIR images for detecting feature changes under the skin and RGB images with their high spatial resolution in non-destructive strawberry detection, a neural network based on the pretrained VGG-19 was used to fuse the images. Using a delicate design, we processed the low- and high-frequency parts of the image separately, preserving the detailed parts of the fused image to a greater extent. The fused images not only contained the high-spatial-resolution RGB images but also contained richer spectral information. This also effectively avoided the regional limitations of the mean relative reflectance method in each ROI, effectively improving the objectivity of the detection results. Compared with the current mainstream algorithms, the proposed fusion method achieved the best structural similarity and information content and minimal noise introduction, having 5.42% and 3.61% higher performance levels than the RGB-only-based and NIR-only-based groups, respectively.

In addition, in order to fit multi-detection objects and multi-tasks, the proposed model can also modify the weights of the RGB and NIR parts. In order to cope with the different sensitivity levels of information sources brought by different detection objects, in practical applications, specific weights need to be determined according to the specific situation, and the method proposed in this paper provides this convenience, which can greatly facilitate the multi-task detection task. Although, because of hardware limitations, the computational speed needs to be improved when dealing with a large amount of samples at the same time. However, we are optimistic that faster computing platforms and better optimization algorithms will be developed. These techniques will facilitate real-time model detection and will also allow the model to detect multiple varieties and characteristics of fruits simultaneously.