*2.6. Defect Area Correction*

If the extent of surface defects of expensive fruits is not graded, all fruits containing defects could be sorted as substandard fruits, which would cause serious economic losses to fruit farmers. So, some grading standards of fruits classify fruits according to extent of defects. For example, the number and area of apple surface defects under different grades were restricted in the local standard for apple grading in Beijing, China. Therefore, the number and area of the defects in defective apples needed to be accurately calculated.

The surface of the apple has a certain curvature because of the similarity between an apple and a sphere. When the apple was placed on the separate fruit tray and the industrial camera captured image of the apple, the defects in different areas of the outer surface of the apple would be scaled to varying degrees. Therefore, the actual area of defect might be different from the region of defect in apple images. Thus, projection method was presented to provide a solution for building the relationship between actual area of defect and defect region in apple images.

In order to eliminate the influence of surface curvature on the defect area in the apple image, it was necessary to correct the number of pixels in the defect area of the apple image. Firstly, apple models of different sizes were obtained by 3D printing, referring to the characteristics of apple surface changes in orchards. There were 12 apple models with horizontal diameter from 68 mm to 90 mm, as shown in Figure 5. In order to establish the corresponding relationship between the number of real pixels and defective pixels in the apple image, a series of black square labels with a side length of 3 mm was printed and pasted on the surface of apple models of different sizes to simulate the change of defect area at different positions.

**Figure 5.** The apple models with labels.

Then, each apple model was put under the camera and the images of the apple models were captured statically. The actual pixel value of the squares and their pixel value in the image were different because of the change of curvature and the different distance from each square to the center of the apple. Therefore, it was necessary to establish the function relationship between the three variables, namely the number Z of real pixels, the distance d from the defect region to the center of the apple and the r representing the radius of the apple. The function relationship could be represented as Z = *F*(*d*,*r*). In order to obtain the expression of the function, the number of pixels in the defect area of apple model image at different positions was recorded manually for apple models with different sizes. Then, the dataset (*dw*, *rw*, *zw*) corresponding to the three variables was generated.

So, given a dataset (*dw*, *rw*, *zw*), *w* = 1, 2, 3,..., *n*. The bivariate polynomial function *F*(*d*,*r*) based on the dataset could be expressed as:

$$F(d,r) = \sum\_{i \bar{j} = 1,1}^{p,q} g\_{i\bar{j}} d^{i-1} r^{\bar{j}-1} = \sum\_{i=1}^{p} \sum\_{j=1}^{q} g\_{i\bar{j}} d^{i-1} r^{\bar{j}-1} \tag{1}$$

Let

*d* = ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 1 *d d*2 . . . *dp* ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ ,*r* = ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣ 1 *r r*2 . . . *rp* ⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦ , G = ⎡ ⎢ ⎣ *g*<sup>11</sup> ··· *g*1*<sup>q</sup>* . . . ... . . . *gp*<sup>1</sup> ··· *gpq* ⎤ ⎥ <sup>⎦</sup> (2)

Then, the function could be expressed as

$$F(d, r) = d^{\mathrm{T}} \mathrm{Gr} \tag{3}$$

The goal of fitting was to obtain the parameter matrix G. To obtain the parameter matrix G, a multivariate function with respect to the parameter *gij* was constructed:

$$L\left(\mathcal{g}\_{11\prime}\cdots\mathcal{g}\_{pq}\right) = \sum\_{w=1}^{n} \left[ F(d\_{w\prime}r\_w) - z\_w \right]^2 = \sum\_{w=1}^{n} \left( \sum\_{i=1}^{p} \sum\_{j=1}^{q} \mathcal{g}\_{ij} d^{i-1}r^{j-1} - z\_w \right)^2 \tag{4}$$

The point (*g11*,..., *gpq*) was the minimum point of the multivariate function L (*g11*,..., *gpq*), and *zw* was the number of actual pixels, so the point (*g11*,..., *gpq*) must satisfy the equation:

$$\frac{\partial L}{\partial \mathbf{g}\_{ij}} = 2 \sum\_{w=1}^{n} \left[ d\_{w}^{i-1} r\_{w}^{j-1} F(d\_{w}, r\_{w}) - d\_{w}^{i-1} r\_{w}^{j-1} z\_{w} \right] = 0 \tag{5}$$

So, the following equation could be obtained:

$$\sum\_{w=1}^{n} d\_{w}^{i-1} r\_{w}^{j-1} F(d\_{w}, r\_{w}) = \sum\_{w=1}^{n} d\_{w}^{i-1} r\_{w}^{j-1} z\_{w} \tag{6}$$

According to Equation (1), there were:

$$\sum\_{w=1}^{n} d\_{w}^{i-1} r\_{w}^{j-1} z\_{w} \sum\_{a=1}^{p} \sum\_{\beta=1}^{q} g\_{a\beta} d\_{w}^{w-1} r\_{w}^{\beta-1} = \sum\_{w=1}^{n} d\_{w}^{i-1} r\_{w}^{j-1} z\_{w} \tag{7}$$

$$\sum\_{\substack{a,\beta=1,1\\a\beta=1,1}}^{p,q} \left[ \mathcal{g}\_{a\beta} \sum\_{w=1}^{n} \left( d\_{w}^{a-1} r\_{w}^{\beta-1} d\_{w}^{i-1} r\_{w}^{j-1} \right) \right] d\_{w}^{i-1} r\_{w}^{j-1} z\_{w} = \sum\_{w=1}^{n} d\_{w}^{i-1} r\_{w}^{j-1} z\_{w} \tag{8}$$

Let *uαβ*(*i*, *j*) = ∑*<sup>n</sup> <sup>w</sup>*=1(*<sup>d</sup> <sup>α</sup>*−<sup>1</sup> *<sup>w</sup> r <sup>β</sup>*−<sup>1</sup> *<sup>w</sup> <sup>d</sup>i*−<sup>1</sup> *<sup>w</sup> <sup>r</sup> j*−1 *w* and *v*(*i*, *j*) = ∑*<sup>n</sup> <sup>w</sup>*=<sup>1</sup> *<sup>d</sup>i*−<sup>1</sup> *<sup>w</sup> <sup>r</sup> <sup>j</sup>*−<sup>1</sup> *<sup>w</sup> zw* So, Equation (8) can be rewritten in matrix form:

$$
\begin{bmatrix}
u\_{11}(1,1) & \cdots & u\_{pq}(1,1) \\
\vdots & \ddots & \vdots \\
u\_{11}(p,q) & \cdots & u\_{pq}(p,q)
\end{bmatrix}
\begin{bmatrix}
\mathcal{g}\_{11} \\
\vdots \\
\mathcal{g}\_{pq}
\end{bmatrix} = 
\begin{bmatrix}
v(1,1) \\
\vdots \\
v(p,q)
\end{bmatrix} \tag{9}
$$

Equation (9) could be rewritten as the form Ug = V, where U is matrix with *pq* × *pq*, and V is a column vector with length *pq*. The column vector g could be calculated. Then, g was transformed into the parameter matrix G. So, the function *F*(*d*,*r*) could be determined using matrix G.

The object distance between the apple and the camera lens would change due to the different size of the apple, and it would affect the conversion between the real pixels and real area corresponding to the real pixels. In order to determine the corresponding relationship between the number of defective pixels and the actual defect areas under different apple sizes, every model apple with a certain size was used to determine the calibration coefficient c(*r*) between the number of pixels and the real areas S, where *r* was the radius of apple. For c(*r*)∈C, C was composed of 20 calibration coefficients. The final area projection equation was:

$$\mathbf{S} = \mathbf{c}(r) \sum\_{i \in S\_1} F(d\_{i'}, r) \tag{10}$$

where *i* represents the pixel located in the defect region *S*<sup>I</sup> in the apple image.

According to Equation (1) to Equation (9), the actual number of pixels corresponding to the defect in the image could be determined. Then, the actual area of the defect could be obtained according to Equation (10). So, the grade of defective apple could be determined according to the apple grading standard.
