A Study on Hyperspectral Apple Bruise Area Prediction Based on Spectral Imaging

Abstract

Achieving fast and accurate prediction of the fruit mechanical damage area is important to improve the accuracy and efficiency of apple quality grading. In this paper, the spectral data of all samples in the wavelength range from 376 to 1011 nm were collected, the sample set was divided by the physicochemical coeval distance method, and the spectral preprocessing methods were evaluated by establishing a full-wavelength artificial neural network model. The wavelength selection of spectral data was performed by competitive adaptive reweighted sampling, L1 parameter method, and the Pearson correlation coefficient method, and the partial least squares, artificial neural network, and support vector machine (Gaussian kernel) prediction models were established to predict the fruit bruise area size. The surface fitting was performed using the actual apple bruise area, and the regression surface equation of the damage time and damage height of the fruit was established. The results showed that (1) the preprocessing method of first-order difference + SG smoothing can make the prediction model more accurate; (2) the CARS-ANN prediction model has better prediction performance and higher operation efficiency, with the prediction set root mean square error of prediction and R-value of 0.1150 and 0.8675, respectively; (3) the sparrow search algorithm was used to optimize the model, which improved the accuracy of the prediction model. The root mean square error of prediction reached 0.0743 and The R-value reached 0.9739. (4) The relationship between spectral information, bruise area, damage time, and damage degree was obtained by combining the establishment of the fitted surface of the apple bruise area with the prediction model. This study is of application and extension value for the rapid nondestructive prediction of fruit bruise area.

1. Introduction

Apples are prone to mild bruises during picking, packing, and transportation, and early mild bruises are characterized by slight depressions, no juice spillage, and difficult recognition by the naked eye. In China, the quality of fruit sorting is mainly based on manual and machine vision sorting systems, which result in a significant waste of human and financial resources [1,2,3,4,5]. Traditional machine vision techniques can be used to detect surface defects and injuries of apples, but the recognition accuracy is not high for early mechanical damage or minor bruises [6,7,8,9,10].

In recent years, with the development of science and technology, optical sensing and imaging techniques have become effective tools for nondestructive testing and assessing the quality and safety of agricultural products. Hyperspectroscopy has been increasingly applied to fruit quality assessment due to its advantages of rich information, no damage to samples, and high detection accuracy [11,12,13,14,15,16,17,18,19,20]. In 2005, Qin et al. analyzed the spectral region from 692 to 856 nm to optimally identify the kernels of sour cherries [21] and used a portable hyperspectral imaging device to detect citrus rot in the 400–900 nm band [22].In 2010, Zhao et al. used hyperspectral image data obtained in the 408–1117 nm spectral range to detect minor abrasions of “crystal” pears and obtained better-sorting results by comparing four algorithms for minor abrasions of pears [23]. Rivera V et al. compared five methods for segmenting mangoes at selected wavelengths in multispectral images and finally selected k-NN for the classification of intact and bruised pixels The correct rate reached 98% [24]. Qibing Zhu et al. developed a partial least squares model using spectral scattering images for each impact energy level as well as pooled data to predict the susceptibility of apples to bruising with high prediction accuracy [11]. In 2014, Zhang Baohua et al. acquired spectral images in the 400–1000 nm band and used PC3 images after quadratic principal component analysis for apple bruise region detection with an accuracy of 95.8% [25]. In 2017, Shuxiang Fan et al. used the CARS-LS-SVM model as well as spectral band ratio images to detect early internal damage in blueberries, and the results showed that hyperspectral images can detect internal bruises in blueberries within 30 min after the mechanical impact [5]. The PLS-DA method was used to rapidly distinguish the ripening stage of the bruised oil tea fruit samples by using a hyperspectral imaging system. In 2021, Ye Sun et al. conducted a collision test on tomatoes and collected hyperspectral images of bruised tomatoes and used a partial least squares discriminant analysis model to classify bruised tomatoes with an overall classification accuracy of 90.93% [26]. In 2022, Zhang Zhen et al. analyzed the performance of spectral indices in the wavelength range from 350 to 2500 nm in determining apple hardness and used the dual-band combination method developed by them to identify effective spectral features as the best indicator of apple hardness with good results [27]. Using a hyperspectral imaging system, Jiang Huazhong et al. developed a PLS-DA model to rapidly zone the ripening stage of split oil tea fruit samples [28]. Yuan R. et al. developed the A-raw-iVISSA (interval variable iterative space shrinkage approach)-PLS-DA model by collecting reflectance, absorbance, and Kubelka–Monk spectral information of Lingwu long dates to predict their healing sensitivity with high accuracy [12].

Although more and more experts and scholars have used hyperspectral imaging technology for fruit quality evaluation in recent years and achieved many good results, relatively few studies have been conducted on the variation law of the fruit bruise area, and relatively few studies have been conducted on the influence law of damage degree and damage time on the bruise area of fruits. Therefore, this paper takes apples as an example and establishes a model for the change of the apple bruise area by acquiring and analyzing the hyperspectral images of apples to make a relatively accurate assessment of the damage time and damage degree of apples.

2. Materials and Methods

2.1. Sample Preparation

The test samples were all red Fuji apples, purchased on 20 March 2022, from Chengdao Supermarket in Tai’an, Shandong Province, and all apples were 8–9 cm in diameter with no obvious bruises. The hardness was measured as approximately 7.5 kg/cm² using a Japanese triple hardness tester model LX-A, and the sugar content was measured as approximately 16.1% using an RY-107 model Toughness Brix tester. After the apples were delivered to the laboratory, the apples were numbered at room temperature (approximately 20 °C) and stored until the end of the experiment.

2.2. Damage Test and Construction of Hyperspectral Imaging System

The apple collision test bench was established as shown in Figure 1a. Apples were attached to the electromagnet with metal spheres of 25 mm in diameter, and the slider was controlled to stay at 20, 30, 40, 50, 60, and 70 cm from the top of the fruit, respectively, and the electromagnet was made to lose its magnetic force by power failure, and the metal spheres fell freely causing apple bruise. The hyperspectral image information was collected one by one when the samples were just bruised and when the bruised apples were left for 1 h, 6 h, 24 h, 72 h, and 120 h. To ensure the reliability of the test data, each group was repeated eight times.

The system used to acquire the spectra is shown in Figure 1b. The camera model is SOC710, and the equipment used for illumination is two 100 w adjustable tungsten halogen lamps; the spectral range acquired by this camera is 376–1011 nm, the resolution is 4.68 nm, and the number of bands is 128.

2.3. Sample Bruise Area Collection

The principal component analysis is a method for reducing the dimensionality of data. It regroups the original variables into a new set of mutually uncorrelated composite variables. The method removes a few less aggregated variables from the original data and retains as much information as possible about the data, depending on the actual needs.

As shown in Figure 2, the collected spectra were transformed by principal component analysis (PCA), the PC4 images with obvious bruised areas were selected, and the bruised areas were segmented by the maximum inter-class difference (Otsu) algorithm. Moreover, the bruised area of the fruit was calculated based on the actual reference ruler and image scale.

3. Results

3.1. Characteristics of Apple Spectral Curves

The spectral images were corrected and imported into the envi5.3 software. Through multiple validations, a marker size of 25 × 25 pixels was selected at the location of the bruise, and its average spectral curve was obtained. The average spectral profile is presented in Figure 3.

$$I=\frac{I_R-I_D}{I_w-I_D}$$

(1)

In order to eliminate electronic noise generated by operation, environmental influences, etc., the acquired spectral image needs to be corrected according to Equation (1). The I_R in the formula is the original hyperspectral image, I_W is the spectral image acquired from the standard whiteboard, I_D is the spectral image in full black, and I refers to the corrected spectral image.

As can be seen in Figure 3, the overall spectral profiles of the apple bruise sites at different times and degrees of damage are relatively similar, with two valleys near 680 nm and 970 nm, which are caused by changes in chlorophyll and changes in water content in the fruit. With the different heights of damage and time after damage, the spectral characteristics of the bruise parts of the fruit were changing, which may be due to the fact that the collision of the apple caused changes in the surface and internal cellular tissue of the fruit, resulting in weakened light scattering [29].

3.2. Sample Set Delineation

As shown in Equation (2), with the bruise area size as the Y variable and the spectral data as the X variable, the sample set partitioning based on the joint X-Y distance (SPXY) algorithm was used to divide the apple spectral data set into a training set, validation set, and test set in the ratio of 4:1:1.

$$\begin{array}{c}{d}_x(p,q)=\sqrt{{\displaystyle \sum }_{j=1}^N{\left[x_p(j)-x_q(j)\right]}^2;p,q\in [1,N]}\\ d_y(p,q)=\sqrt{{\left(y_p-y_q\right)}^2}=\left|y_p-y_q\right|;p,q\in [1,N]\\ d_{xy}(p,q)=\frac{d_x(p,q)}{{\max }_{p,q\in [1,N]}{d}_x(p,q)}+\frac{d_y(p,q)}{{\max }_{p,q\in [1,N]}{d}_y(p,q)};p,q\in [1,N]\end{array}$$

(2)

where,

• x: spectral data
• y: bruise area
• d: Euclidean distance between two samples
• N: number of samples

Calculating the inter-sample distances can retain the characterization samples with the largest features, improve the representativeness of the samples, and facilitate the stability of the model. The results of the data set division are shown in

Table 1. Area of apple bruises.

Sample Set	Number of Samples	Maximum Value/cm2	Minimum Value/cm2	Mean Value/cm2	Standard Deviation/cm2
training set	96	1.6672	0.5066	0.9599	0.3270
validation set	48	1.6622	0.5116	0.9547	0.3304
text set	48	1.6509	0.5203	0.9774	0.3245

.

3.3. Spectral Pretreatment

A single preprocessing method often cannot meet the model accuracy requirements. Therefore, in this paper, the common spectral preprocessing methods are divided into three categories: the first category is for baseline correction processing, including first-order differential and second-order differential; the second category is for eliminating the effect of scattering on the spectrum caused by different particle size and distribution uniformity, including multiplicative scatter correction (MSC) and standard normal variate transformation (SNV); the third category is for eliminating random noise and optimizing the signal, mainly by Savitzky–Golay (SG)and Gaussian Smoothing (GS). The above three types of pre-processing methods are combined two by two to process the original signal, and an ANN model is built to evaluate its accuracy. The results are shown in

Table 2. Full-wavelength ANN prediction models for different pretreatment methods.

Pre-Processing Method	Validation Set		Text Set
	R	RMSE	R	RMSE
RAW	0.8015	0.2554	0.9101	0.2754
First-order differential +MSC	0.8554	0.2756	0.8531	0.2883
First-order differential +SNV	0.8775	0.2004	0.8466	0.2210
First-order differential +SG	0.9028	0.2005	0.9014	0.2045
First-order differential +GS	0.8467	0.2320	0.8399	0.2487
Second difference +MSC	0.8051	0.2694	0.8002	0.2808
Second difference +SNV	0.8324	0.2406	0.8287	0.2536
Second difference +SG	0.8214	0.2611	0.8157	0.2628
Second difference +GS	0.8525	0.2359	0.8453	0.2498
MSC + SG	0.8373	0.2525	0.8341	0.2547
MSC + GS	0.7145	0.5104	0.7116	0.5112
SNV + SG	0.8464	0.2499	0.8338	0.2501
SNV + GS	0.7282	0.3979	0.7245	0.5025

.

As can be seen from Table 2, the performance of the pre-processed spectral prediction is mostly higher than the original unprocessed spectral information, among which the model accuracy after first-order difference + SG processing is the highest, with the correlation coefficients of the calibration set and prediction set reaching 0.9028 and 0.9014, and the root mean square error of 0.2005 and 0.2045. The spectral information map after processing is shown in Figure 4. Therefore, the first-order difference + SG processing was subsequently used as a pre-processing method to participate in the subsequent modeling.

3.4. Feature Band Selection

The hyperspectral data collected in the experiment are all 696 × 520 × 128 pixels, and the data are lengthy and have many wavelengths unrelated to the detection target. Therefore, it is necessary to extract the feature wavelengths related to the detection target for subsequent modeling analysis.

3.4.1. Extraction of Feature Wavelengths Using Competitive Adaptive Reweighted Sampling (CARS)

Competitive adaptive reweighted sampling was used to extract the feature wavelengths for building the partial least squares regression model.

Figure 5 shows the schematic diagram of the sample image after processing by the CARS algorithm. As shown in Figure 5a, the number of wavelengths retained by the algorithm decreases as the number of samples increases, and the degree of variation in the number of retained wavelengths also decreases. Figure 5b plots the variation of RMSECV after 5-fold cross-validation, and the minimum RMSECV is 0.2177 when the number of samples is 5–32, and it continues to increase as the number of samples increases the RMSECV value. Figure 5c depicts how the regression coefficients for each variable change with iteration, and the star-shaped vertical line indicates the best model with the lowest RMSECV. Ultimately, the samples were subjected to wavelength screening by CARS, and a total of 11 wavelengths were screened out.

3.4.2. Extraction of Characteristic Wavelengths by Pearson Correlation Coefficient Method

The Pearson correlation coefficient method is a method for measuring the linear correlation between two variables, and its value range is [−1, 1]: −1 means a completely negative correlation, 1 means a completely positive correlation, and 0 means no linear correlation. The Pearson coefficient of each feature and the target variable is calculated based on Equation (3), and the top k largest features are selected as the final features by ranking according to the absolute value of the magnitude.

$$r=\frac{{{\displaystyle \sum }}_{i=1}^n\left(X_i-\overline{X}\right)\left(Y_i-\overline{Y}\right)}{\sqrt{{{\displaystyle \sum }}_{i=1}^n{\left(X_i-\overline{X}\right)}^2}\sqrt{{{\displaystyle \sum }}_{i=1}^n{\left(Y_i-\stackrel{}{Y}\right)}^2}}$$

(3)

3.4.3. Feature Wavelength Extraction Using the L1 Norm Method

The feature selection method uses Lasso regression, which adds an L1 regularization term to constrain the parameters of the model based on its least squares method and makes a portion of the parameters shrink to zero.

3.5. Modeling

The PLS model obtains the mutually orthogonal feature vectors of the independent and dependent variables by projecting the high-dimensional data into the low-dimensional space, respectively, and then establishes a linear regression relationship between the feature vectors, which can overcome the multicollinearity problem of the data and performs well in small-sample prediction.

Gaussian kernel-based support vector machine (SVM) regression model is a nonlinear support vector regression that uses a Gaussian kernel to map the data from the original space to a high-dimensional feature space and then finds a hyperplane in the feature space that minimizes the sum of the error and regularization terms to accommodate the data with strong generalization capability.

The artificial neural network (ANN) regression modeling method is a method that uses artificial neural networks ANN for regression analysis, which consists of multiple artificial neurons with hierarchical and connection relationships to form a network structure and simulates the signal transmission between neurons utilizing mathematical expressions with strong topology. By choosing the appropriate number of hidden cells and network levels, feedforward networks can approximate nonlinear functions with arbitrary accuracy and are widely used in modeling and control [30].

In this paper, these three methods were applied to build a regression model of spectral data and apple bruise area.

4. Discussion

4.1. ANN Modeling Structure and Parameter Selection

In this paper, the trainlm (L-M) training method with good fitting problem-solving ability was used for model training in the ANN model construction process, and the parameter settings of the activation function and the number of hidden neurons were optimized.

Since the data in the training, validation, and test sets are partitioned by the SPXY algorithm and the data differences are small, the optimal structural parameters can be determined using the root mean squared error (RMSE) of the test set as a result of the model performance.

The hidden layer activation functions logsig and tanh and the output layer activation functions logsig, tanh, and sigmoid were tested. The test results are shown in

Table 3. Activation function optimization.

Combination of Hidden Layer Output Layer Activation Functions	RMSE
logsig/logsig	0.12074
logsig/tanh	0.11503
logsig/sigmoid	0.14382
tanh/logsig	0.11964
tanh/tanh	0.12105
tanh/sigmoid	0.14977

. When the hidden layer activation function is logsig and the output layer activation function is tanh, the modeling effect is better than other combinations.

After several experiments and analyses, the number of hidden layers was set to 3 layers. The number of hidden layer neurons was set from 1 to 10 as the test results are shown in Figure 6, and the RMSE value of the model decreased significantly when the number of hidden neurons increased from 1 to 5. The RMSE value of the model was lowest when the number of hidden neurons exceeded 6. As the number of neurons continued to increase, the RMSE value tended to stabilize. Therefore, in the current study, the optimal number of hidden neurons was set to 6 to simplify the model structure.

4.2. Comparison of Modeling Methods

Three regression models based on apple hyperspectral data and apple bruise area were constructed by PLS, SVM, and ANN. Meanwhile, the prediction results of the apple bruise surface area for different feature selection methods were compared.

Table 4. Comparison of different models for predictive modeling of apple bruise area.

	CARS		L1 Norm		Pearson
	R	RMSE	R	RMSE	R	RMSE
ANN	0.8675	0.1150	0.8410	0.1199	0.8489	0.1555
SVM	0.8547	0.1206	0.8533	0.1238	0.8524	0.1220
PLS	0.7401	0.2177	0.8144	0.1625	0.8014	0.1437

shows the prediction results of the apple bruise area based on different feature selection and regression methods. The closer the correlation coefficient R is to 1, the lower the RMSE is, which indicates that the accuracy of the prediction is higher and the correlation between the actual value and the predicted value is greater. From Table 4, it can be seen that the accuracy of ANN and SVM models is better than that of PLS regression models, and the reason for this is that the relationship between the apple bruise area and hyperspectral information may not be a simple linear relationship.

To further investigate the prediction performance of the three methods for CARS-based feature selection, the result plots of PLS, SVM, and ANN models for apple bruise area prediction are shown in Figure 7a–c, respectively. It can be seen from the figure that all three methods can capture the data variation of the apple bruise area, and most of the predicted values of the bruise area do not differ much from the actual values. Some of the predicted values of the CARS-PLS model have large deviations from the actual values, and the model accuracy is relatively low. The CARS-ANN model has the highest trajectory consistency and the smallest deviation from the actual values compared with the other two models. Combined with Table 4, the CARS-ANN model achieves an R-value of 0.8675 and an RMSE value of 0.1150. The model effect is relatively good, but there is still a possibility of optimization.

4.3. Model Optimization

The sparrow search algorithm (SSA) is a swarm intelligence optimization algorithm proposed by Jiankai Xue et al. inspired by the foraging and anti-predatory behaviors of sparrows [31]. The algorithm has the advantages of strong merit-seeking ability and fast convergence speed. The sparrow flocks foraging process is also a kind of finder–follower model with a detection and warning mechanism superimposed. The individual who finds the better food among the sparrows acts as the finder and the other individuals act as the followers, while a certain percentage of individuals in the population is selected for detection and warning, and if danger is found, the food is abandoned and safety comes first.

The steps are as follows:

• Initialize the sparrow population location and fitness, set the number of producers (PD) = 0.7, the number of sparrows who perceive the danger (SD) = 0.1, the safety value (ST) = 0.6 parameters initial value, start the cycle and derive the current optimal sparrow individual location, and the best fitness value.
• Foraging behavior, update the finder position according to Equation (4) and update the joiner position according to Equation (5).

$$X_{i,j}^{t+1}=\left\{\begin{array}{ll}{X}_{i,j}^t\cdot \exp \left(\frac{-i}{a\cdot N}\right)\hfill & \mathrm{if}{R}_2<ST\hfill \\ X_{i,j}^t+Q\cdot L\hfill & \mathrm{if}{R}_2\ge ST\hfill \end{array}\right.$$

(4)

where,

• Q: random number obeying normal distribution
• L: unit row vector
• N: maximum number of iterations
• a: random number between [0, 1]
• R₂: alarm value
• ST: safety value
• $X_{i,j}^t$: the value of the jth dimension of the ith sparrow at iteration t

3.

Anti-predatory behavior, updating the location of the sparrow population according to Equation (6).

$$X_{I,j}^{t+1}=\left\{\begin{array}{ll}Q\cdot \exp \left(\frac{X_{\mathit{worst}}^t-X_{I,j}^t}{i^2}\right)\hfill & \mathrm{if}i>n/2\hfill \\ X_P^{t+1}+\left|X_{I,j}^t-X_P^{t+1}\right|\cdot A^{+}\cdot L\hfill & \mathrm{otherwise}\hfill \end{array}\right.$$

(5)

where,

• X_worst: lowest adaptive sparrow location
• X_P: optimal position occupied by the producer
• t: number of iterations
• Q: random number obeying normal distribution
• L: unit row vector
• A: A row vector containing only 1 and −1 elements at random,
• A⁺: A^T(AA^T)⁻¹

$$X_{i,j}^{t+1}=\left\{\begin{array}{ll}{X}_{\mathit{best}}^t+\beta \cdot \left|X_{i,j}^t-X_{\mathit{best}}^t\right|\hfill & \mathrm{if}{f}_i>f_{best}\hfill \\ X_{i,j}^t+K\cdot \left(\frac{\left|X_{i,j}^t-X_{\mathit{worst}}^t\right|}{\left(f_i-f_{worst}\right)+\epsilon }\right)\hfill & \mathrm{if}{f}_i=f_{best}\hfill \end{array}\right.$$

(6)

where,

• $X_{\mathit{best}}^t$: current global optimal position
• β: random number that obeys normal distribution
• K: a random number between [−1,1]
• f_i: individual fitness value
• f_worst: worst fitness values
• f_best: the current global best values
• Ɛ: a constant close to 0

4.

Update the historical optimal fitness and repeat steps 2 and 3 and end the loop when the maximum number of iterations is reached. The optimal individual position and fitness values are output. The optimization parameters and threshold process are shown in Figure 8.

The regression coefficients of the training set, validation set, and prediction set before and after the optimization of the ANN model with SSA are shown in Figure 9. Figure 9a shows the before optimization and Figure 9b shows the regression coefficients after optimization. The results show that the correlation coefficient (R) of the modeling set, prediction set, and validation set all reach above 0.96 after the optimization of ANN model parameters and thresholds by the SSA sparrow search algorithm, and the optimization effect is obvious.

4.4. Construction of the Fitted Surface

The mean value of the calculated actual apple bruise area was used as the dependent variable, and the damage height h and damage time t were used as the independent variables to establish the fitting surface using a polynomial fitting method, and the equation of the fitting surface is shown in Equation (7).

$$S=0.22191+0.01449h-0.0136t-1{.9363\times 10}^{-5}{h}^2+1{.36925\times 10}^{-5}{t}^2+6{.58789\times 10}^{-5}\mathit{ht}$$

(7)

where,

S: bruise area

t: damage time

h: ball drop height.

The fitted surface is shown in Figure 10, and the fitted R² = 0.97028, residual sum of squares (RSS) = 0.09562, and the residual sum of squares contour plot are shown in Figure 11, which indicates the high degree of surface fitting. The relationship between the actual bruise area, injury time, and the degree of injury was obtained by the construction of the fitted surface.

4.5. Fruit Minor Mechanical Bruise Area Prediction

In this paper, the PLS, ANN, and SVM prediction models were established using the hyperspectral data of the fruit bruise change area, and SSA was used to optimize the models, and the relationship between the spectral information and the apple bruise area was obtained. Moreover, by constructing the fitted surface equations of the actual area, the damage time, and damage degree, the connection between the spectral information, the bruise area, the damage time, and the damage degree were finally obtained. The specific process is shown in Figure 12.

5. Conclusions

In this paper, we collected the spectral information of the apple bruise area by the collision test. A two-by-two combination of three categories of spectral preprocessing was used to process the raw spectral data, and its performance was evaluated by building a full-band ANN model. The results show that the preprocessing method of first-order differential + SG smoothing can make the prediction model more accurate.

CARS, Pearson, and L1 norm were used for wavelength selection, and the PLS, SVM (Gaussian kernel), and ANN training models were compared, respectively, and the results showed that the accuracy of the CARS + ANN model was higher, with RSME of 0.1150 and R² of 0.8675.

The ANN model was optimized by using SSA, and the optimized model RMSEP reached 0.0743 and the R-value reached 0.9739, which improved the accuracy of the model. The association between spectral information and fruit bruising area was obtained.

The actual bruise area was measured by the collision test, and the resulting data were fitted to the surface to establish the regression surface equations for the fruit damage time and the damage height. The correlation between the spectral information, bruise area, damage time, and damage height was obtained by the prediction model and regression equation establishment. The method is of extended and applied value for rapid nondestructive prediction of fruit injury.