A Specular Highlight Removal Algorithm for Quality Inspection of Fresh Fruits

Abstract

Nondestructive inspection technology based on machine vision can effectively improve the efficiency of fresh fruit quality inspection. However, fruits with smooth skin and less texture are easily affected by specular highlights during the image acquisition, resulting in light spots appearing on the surface of fruits, which severely affects the subsequent quality inspection. Aiming at this issue, we propose a new specular highlight removal algorithm based on multi-band polarization imaging. First of all, we realize real-time image acquisition by designing a new multi-band polarization imager, which can acquire all the spectral and polarization information through single image capture. Then we propose a joint multi-band-polarization characteristic vector constraint to realize the detection of specular highlight, and next we put forward a Max-Min multi-band-polarization differencing scheme combined with an ergodic least-squares separation for specular highlight removal, and finally, the chromaticity consistency regularization is used to compensate the missing details. Experimental results demonstrate that the proposed algorithm can effectively and stably remove the specular highlight and provide more accurate information for subsequent fruit quality inspection. Besides, the comparison of algorithm speed further shows that our proposed algorithm has a good tradeoff between accuracy and complexity.

1. Introduction

Traditional fruit quality inspection methods that rely on human eye detection have disadvantages such as easy fatigue, high labor intensity, long labor time and lack of objectivity, which is far from meeting the ever-increasing market demand. How to realize the fruit quality inspection efficiently and non-destructively has become an important issue in the fruits industry. Non-destructive testing (NDT), as the name suggests, is a technique to evaluate the quality of fruits according to their physical characteristics without destroying the fruit samples [1]. At present, the most used NDT methods for fresh fruits are spectroscopy measurement [2], electronic nose (e-nose) [3], infrared thermography [4], image processing methods based on machine vision [5], dielectric properties [6] and so on.

Machine vision-based methods can evaluate the fruit quality based on characteristic information such as size, color, texture and so on, which is contactless, unified standards, highly efficient and of great importance to save manpower and material resources. Nowadays, this type of testing method is widely applied to the automatic fruit quality testing system, and the main working principle consists of image acquisition and specular highlight removal. When the fruit sample to be tested reach the image acquisition room, the camera captures the fruit image to the computer for processing, and finally the quality inspection result will be given based on the image processing result [7]. The schematic diagram of the automatic quality testing system based on machine vision is shown in Figure 1. To improve the accuracy and efficiency of fruit quality inspection, the researchers have been working to improve the performance of existing machine vision-based image processing algorithms. However, an important issue existing in the image acquisition process is ignored. Owing to the integrated influence of illumination and physical properties of target’s surface, the quality of target image is easily degraded by specular highlight [8]. For fruits with smooth skin and less texture as shown in Figure 2, they are easily affected by specular highlight during the image acquisition, resulting in light spots appearing on the surface of fruits, which severely affects the accuracy of subsequent quality inspection [9,10]. Specifically, the specular highlight areas existing on the surface may be misjudged as damage or fruit russeting and so on, which leads to an inaccurate quality inspection and evaluation results. Aiming at this issue, we need to propose a specular highlight removal algorithm that can be embedded into existing quality inspection systems to improve the accuracy and efficiency.

To solve the problem of specular highlight interference in machine vision-based quality inspection, we propose a new specular highlight removal algorithm based on multi-band polarization imaging, which combines the advantages of single image and multiple images methods. The proposed specular highlight removal algorithm can be integrated into existing quality inspection equipment to achieve effective specular highlight removal and thus ensure the accuracy of quality inspection and evaluation. Our proposed algorithm is composed of two parts, one is image acquisition and the other is specular highlight removal as shown in Figure 1. The image captured by the multi-band polarization imager is a mosaic image including all the spectral and polarization information, so we use a demosaicing algorithm [11] to reconstruct color sub-images with different polarization angles (0°, 45°, 90°). Then the two primary polarization parameters, Degree of Linear Polarization (DoLP) and Angle of Polarization (AoP), can be further calculated by these sub-images based on the following equations:

$$\begin{array}{cc}{S}_0=I_0+I_{90};S_1\hfill & =I_0-I_{90};\hfill \\ S_2\hfill & =2I_{45}-I_0-I_{90}\hfill \end{array}$$

(1)

$$DoLP=\frac{\sqrt{S_1^2+S_2^2}}{S_0};AoP=\frac{1}{2}arctan\left(\frac{S_2}{S_1}\right)$$

(2)

In the second part, we propose a specular highlight removal method based on multi-band polarization imaging. Firstly, we propose a joint multi-band-polarization characteristic vector constraint to realize the detection of specular highlights. Then, we put forward a Max-Min multi-band-polarization differencing scheme; meanwhile, an ergodic least-squares separation is combined to remove the specular highlight. Finally, the chromaticity consistency regularization is utilized to compensate for the missing details. Experimental results show that both visual effects and quantitative evaluation of our proposed algorithm are better.

2. Related Work

When incident light passes through the surface of a medium, there are two different reflection effects, diffuse reflection and specular reflection [12]. Under ideal lighting conditions, the surface of an object is often assumed to be a Lambertian surface with only diffuse reflection. The surface of the object presents the same brightness from all viewpoints and the brightness of a certain point on the surface of the object does not vary with the observation angles. Diffuse reflection is related to the surface material and reflects the physical and chemical properties of the target surface (such as roughness) [13], while, in fact, not only diffuse reflection but also specular reflection occurs on the target surface, resulting in specular highlight with high intensity existing on the target surface during the observation process. Furthermore, unsurprisingly, the specular highlight would conceal useful image features such as colors and textures. Different from diffuse reflection, specular highlight is related to the light source and viewpoints and varies with the direction and intensity of the light source, which effectively reflects the information of the light source [14].

For specular highlight removal, numerous methods have been proposed and they can be classified into single image methods and multiple images methods according to the number of input images. In this section, we review the previous specular highlight removal methods.

Single image specular highlight removal algorithms. The earliest method dates back to 1996, when Bajcsy et al. [15] proposed a specular highlight removal algorithm based on color segmentation, but later it was proven to be unable to handle targets with complex textures. Tan et al. [16] proposed a fully automatic method based on chromaticity analysis in 2005, they utilize the difference between specular and diffuse pixels in their proposed pseudo-specular-free (PSF) image to remove the specular highlight. To reduce the high computational cost, Yang et al. [17] accelerated the algorithm [16] by employing joint bilateral filtering. Similarly, Mallick et al. [18] proposed a specular highlight removal algorithm by evolving a partial differential Equation (PDE) that iteratively erodes the specular component at each pixel. Shen et al. [19] proposed a simple method to separate specular highlights in a color image based on the error analysis of chromaticity and appropriate selection of body color for each pixel. Yoon et al. [20] first proposed a specular-free two-band image, which was a specular-invariant color image representation. Furthermore, the highlight removal was achieved by comparing the local ratios of each pixel and these ratios were made equal in an iterative framework. Shen and Zheng [21] proposed a real-time highlight removal method, which works by pseudo-chromatic space analysis, clustering and estimation of intensity ratios in mixed specular-diffuse regions. This method works in a pixel-wise manner, without specular pixel identification and any local interaction. Based on the concept of dark channel prior, Kim et al. [22] proposed an optimization framework to achieve specular highlight removal. A new framework that combines the non-negative matrix factorization (NNMF) with a sparsity constraint for highlight removal was proposed by Akashi and Okatani [23]. Suo et al. [24] proposed a fast highlight removal algorithm based on an $L_2$ chromaticity definition and extended dichromatic reflection model (DRM). Ren et al. [25] proposed an algorithm based on color lines. Firstly, a modified nearest neighbor technique is used to cluster the image. Then searching along the radius in polar coordinates to recover the diffuse coefficient. Finally, both the color of illumination and diffuse reflection can be recovered. Fu et al. [26] proposed a specular highlight removal method for real-world images. They observed that for real-world images, the specular highlight often has a small size and sparse distribution; and the diffuse image can be represented by a linear combination of a small number of basis colors with the sparse encoding coefficients. So, they designed an optimization framework that can simultaneously estimate the diffuse and specular highlight images. Nguyen-Do-Trong et al. [10] excluded the specular highlight from the acquired hyperspectral reflectance images by implementing the cross-polarization to block the specular reflection, and they evaluated the cross-polarization approach in a line-scanning hyperspectral reflectance imaging system for the first time. Boyer [27] also used a cross-polarization imaging system to reduce the specular artifacts. Wen et al. [28] developed a polarization-guided model to generate a polarization chromaticity image and then reformulated the problem into a global energy function based on the proposed model. Finally, they optimized the global energy function by using the ADMM strategy to realize specular reflection separation.

Input image acquisition is simple and fast for single image methods since there is no need for special illumination or equipment, while the traditional single image methods have a common problem: they realize the removal of specular highlights based on particular prior information or assumptions. When the predetermined conditions are not satisfied or the image scenes change, it may lead to color discontinuity, edge distortion or a loss of structural and textural information. Most of the newer methods on specular highlight removal are based on deep learning. Although this kind of method has made significant progress [29,30], there are still some problems. First, deep-learning-based algorithms suffer from generic problems such as high computational effort, high hardware cost, and complex model design. Second, deep learning-based methods are usually trained on synthetic data or very few real data, and thus the desired results may not be obtained for real images due to the domain gap between training and test images.

Multiple images specular highlight removal algorithms. The prominent disadvantage of multiple images methods is that the imaging equipment or illumination conditions are complicated. However, multiple images algorithms have more accurate specular highlight removal results since they have richer input information. Nayer [31,32] combined color space analysis with projection constraints on the specular reflection component of the target polarized image to separate specular highlight. Based on the assumption that specular highlight is statistically uncorrelated with the diffuse component, Umeyama [33] first obtained the intensity difference between two images with different polarization angles to discriminate the specular region and then used independent component analysis (ICA) to estimate the specular highlight component in the specular region. By placing the polarization filter in front of the sensor, Wang [34] produced results with less color distortion. Yang [35] used the polarization imaging technique to detect bruises on nectarines. The polarized image can effectively suppress the specular highlight on the smooth surface of nectarine and overcome the interference of dark color during detection. A lightweight network (ResNet-G18) that integrate ResNet-18 and Ghost bottleneck was constructed to achieve classification and damage detection of nectarines. Lin et al. [36] proposed an algorithm based on color analysis and multi-baseline stereo vision, which can estimate the separation and the true depth of specular reflections at the same time. Lin et al. [37] proposed a method based on the neutral interface reflection model for separating specular reflection components in color images. Different from most previous methods, this approach did not assume any dependencies among pixels, such as regionally uniform surface reflectance. The image is a superposition of a transmitted layer and a reflected layer when it is recorded through a transparent medium (e.g., glass), Guo et al. [38] proposed a method to separate the two layers from multiple images by designing a novel Augmented Lagrangian Multiplier based algorithm.

3. Methodology

Our specular highlight removal algorithm consists of two parts as shown in Figure 3, one is image acquisition and the other is specular highlight removal. The task of the first part is to quickly acquire all the spectral and polarization information of the target, so we design a multi-band polarization imager with a new configuration here to realize highly efficient and multi-dimensional image acquisition. The second part’s task is to remove the specular highlight from images accurately, so we propose a new specular highlight removal method based on multi-band polarization imaging.

3.1. Images Acquisition Based on Multiband Polarization Imager

The imaging technique combining spectroscopy and polarization has been applied to fruit quality inspection [39]. With the rapid development of micro-nano manufacturing, array-based multi-band polarization imaging technology has gradually become a research focus in recent years. This technology is the product of a combination of intensity, polarization and spectral imaging technologies, which can obtain spatial, polarization and spectral information of the target at the same time through single image capture, thus realizing multi-dimension detection of the target.

The existing literature focuses on the configuration design and performance analysis of polarization arrays [40,41] or multi-band arrays [42] alone. In recent years, new array-based multi-band polarization imager has been produced, such as the PHX050S camera from Lucid Vision Labs, which can capture multi-band and polarization information of the scene simultaneously. Inspired by this, we design a multi-band polarization imager with a new configuration as shown in Figure 3a in the image acquisition part. The imager mainly consists of a multi-band polarization imaging array and a CMOS. Different colors indicate the selection of different wavebands and different grating orientations indicate the selection of different polarization angles. Since our subsequent proposed specular removal method requires information in visible light wavebands (R, G, B wavebands) and three polarization angles (0°, 45°, 90°), So we combine the “Quasi-Bayer” polarization pattern [43] and “Bayer” color filter array [44] together to constitute a new multi-band polarization imager as shown in Figure 3a. In this way, spectral information in R, G, B bands and polarization information in 0°, 45°, 90° can be acquired at the same time through single image capture.

As shown in Figure 3b, the captured image of our multi-band polarization imager is a multi-band polarization mosaic image, so the chromatic polarization demosaicking network (CPDNet) proposed by Wen [11] is applied here to reconstruct our required spectral and polarization information. Each pixel in the multi-band polarization array records only 1 out of 9 necessary intensity measurements. For each channel, the relationship between mosaic image Y and three full-resolution multi-band polarization images X can be formulated as:

$$Y=D_{\theta }X$$

(3)

where $D_{\theta }$ represents the down-sampled matrix, with

$$\theta \in \left\{R{0}^{\circ },R{45}^{\circ },R{90}^{\circ },G{0}^{\circ },G{45}^{\circ },G{90}^{\circ },B{0}^{\circ },B{45}^{\circ },B{90}^{\circ }\right\}$$

(4)

The goal of CPDNet is to learn the mapping function $F\left(Y\right)=X$. The input of the CPDNet is the multi-band polarization mosaic image that can be denoted as $Y\in I^{m\times n}$, where m and n indicate the number of rows and columns of the input image. Accordingly, the ground-truth image can be expressed as $X\in I^{m\times n\times 9}$.

By using the CPDNet, we can reconstruct a color polarization image in three angles from a multi-band polarization mosaic image as shown in Figure 3d. The specific demosaicking process is not described in detail here, readers can refer to [11] for more information.

3.2. Specular Highlight Removal

According to the dichromatic reflection model proposed by Shafer [13], the reflected light can be regarded as a combination of diffuse reflection and specular reflection, which can be described mathematically as:

$$\begin{array}{cc}I\left(x\right)\hfill & =I_d\left(x\right)+I_s\left(x\right)\hfill \\ \hfill & ={\rho }_d\left(x\right)D\left(x\right)+{\rho }_s\left(x\right)S\left(x\right)\hfill \end{array}$$

(5)

where $I_d\left(x\right)$ is diffuse reflection component and $I_s\left(x\right)$ is specular highlight component; ${\rho }_d$ and ${\rho }_s$ represent weighting factors of diffuse reflection and specular reflection, respectively; and $D\left(x\right)={\int }_{\mathsf{\Omega }}^{}H\left(\lambda ,x\right)E\left(\lambda \right)q\left(\lambda \right)d\lambda$, $S\left(x\right)={\int }_{\mathsf{\Omega }}^{}E\left(\lambda \right)q\left(\lambda \right)d\lambda$, where $\lambda$ is wavelength, $\mathsf{\Omega }$ is the range of wavebands. $q=\left\{q_r,q_g,q_b\right\}$ is the camera induction function, H represents spectral reflectance, and E denotes the spectral power distribution function of light source.

In natural light imaging environment (namely, the light source is unpolarized light), the diffuse reflection component has a weak polarization effect due to multiple scattering processes, so the diffuse reflection component can be approximately regarded as unpolarized light, while the specular highlight reflects only once on the target surface so that it has strong polarization effect, thus the specular highlight can be approximately regarded as partial linear polarized light. As depicted in [45], the intensity of the captured target image varies as the cosine function of the rotation angle $\phi ={\theta }_{pol}$ of the polarizer. The intensity of captured image can be expressed as:

$$I\left(\phi \right)=\frac{I_{max}+I_{min}}{2}+\frac{I_{max}-I_{min}}{2}cos(2\phi -2\alpha )$$

(6)

where $I_{max}$ and $I_{min}$ represent the maximum and minimum intensities of the captured image, respectively, when rotating the polarizer. $\alpha$ is the polarization phase angle, and here in our study, $\theta \in \left\{0^{\circ },{45}^{\circ },{90}^{\circ }\right\}$.

The intensity of diffuse reflection does not change with the rotation of the polarizer because the diffuse reflection component can be approximated to unpolarized, so we have:

$$I_d=I_{min}$$

(7)

For the varying specular highlight component, its intensity can be considered as the sum of a specular constant $I_{sc}$ and a cosine variable $I_{sv}cos(2\phi -2\alpha )$:

$$I_s=I_{sc}+I_{sv}cos(2\phi -2\alpha )$$

(8)

So, the intensity of the captured image in different wavebands can be described as:

$$I_{\lambda }\left(\phi \right)=I_{\lambda ,d}+\frac{I_{\lambda ,s}}{2}+\frac{I_{\lambda ,s}}{2}cos2\phi sin2\alpha +\frac{I_{\lambda ,s}}{2}sin2\phi cos2\alpha$$

(9)

$$I_{\lambda }\left(\phi \right)=I_{\lambda ,d}+f\left(\phi \right)I_{\lambda ,s}$$

(10)

where $f\left(\phi \right)$ is weighting function of highlight component ($0\le f\left(\phi \right)\le 1$), which is only related to polarizer’s rotation angle $\phi$ while has nothing to do with waveband $\lambda$. Furthermore, $\lambda \in \left\{r,g,b\right\}$.

In a conclusion, the specular highlight has different polarization states in different wavebands. Therefore, the diffuse reflection component and highlight component can be distinguished effectively through comprehensive utilization of the polarization information in different wavebands.

3.2.1. Highlight Detection Based on Multiband Polarization

We conduct a statistical analysis to further analyze the differences in spectral and polarization characteristics between specular highlight and diffuse reflection areas intuitively. Here 100 images affected by specular highlights are collected as the data for statistical analysis. Firstly, we manually segment these image data to obtain the two independent areas: for each image data, two small patches that consist of $5\times 5$ pixels are extracted from the diffuse reflection area and specular highlight areas, respectively. Thus, we obtain two sets of images, one is specular highlight patches and the other is diffuse reflection patches. Then we compute the DoLP (Degree of Linear Polarization) and grayscale values of each pixel in the R, G, and B bands, and the average values are shown in Figure 4. It can be clearly seen that the DoLP values of pixels in the diffuse reflection area are quite low while the specular highlight pixels have higher DoLP. Besides, for different wavebands, the distributions of DoLP are different either. Similarly, the grayscale value of the diffuse reflection area is much lower than that of the specular highlight area and the distributions are also different in each waveband. Therefore, the specular highlight and diffuse reflection components can be distinguished accurately by analyzing the differences in intensity, spectral and polarization characteristics, so that effective specular highlight detection can be achieved.

Based on the above analysis, we define the joint multi-band-polarization characteristic vector as follows:

$$V\left(x\right)={\left[Dol{p}_{\lambda }\left(x\right),l\left(x\right)\right]}^T$$

(11)

$$l\left(x\right)=0.299r\left(x\right)+0.587g\left(x\right)+0.114b\left(x\right)$$

(12)

where $Dolp$ is the degree of linear polarization, $\lambda \in \left\{r,g,b\right\}$ and $r,g,b$ represents intensity values of three visible bands, respectively. Additionally, it should be noted here that the ratios of $r,g,b$ are fixed weight coefficients (0.299 for r rand, 0.587 for g rand and 0.114 for b rand), since we use the function “rgb2gray” in MATLAB to transform the color image into a gray-scale image. The values of $Dol{p}_{\lambda }$ range from 0 to 1, and both $Dol{p}_{\lambda }$ and $l\left(x\right)$ are in the same vector $V\left(x\right)$, so $l\left(x\right)$ is normalized for consistency. Namely, the possible values of $l\left(x\right)$ range from 0 to 1 as well.

The specular highlight area and the diffuse reflection area can be detected by effective constraints based on the joint multi-band-polarization characteristic vector. For the target image under the influence of specular highlight, its pixel x meets the following constraints:

$$x\in \left\{\begin{array}{cc}R\left(D\right),& if{∥V\left(x\right)-V_0∥}_2^2<\epsilon \\ R\left(S\right),& if{∥V\left(x\right)-V_0∥}_2^2>\epsilon \end{array}\right.$$

(13)

where $R\left(D\right)$ and $R\left(S\right)$ represent diffuse reflection area and specular highlight area, respectively, $V_0$ denotes the global threshold vector, $\epsilon$ is a tiny positive constant.

To facilitate subsequent highlight removal, a binary mask image is generated for the detection results of pixel x, and its expression is:

$$mask\left(x\right)=\left\{\begin{array}{cc}1,& ifx\in R\left(S\right)\\ 0,& ifx\in R\left(D\right)\end{array}\right.$$

(14)

Based on the joint multi-band-polarization characteristic vector constraint, for the target image shown in Figure 3, The values of $V_0$ and $\epsilon$ are determined by the following analysis. Figure 5a is the target image, for the marked area of the target image in (a), the distribution statistical characteristics of Dolp, grayscale and luminance in R, G and B bands are shown in Figure 5b–e, where the X-axis is the pixel position, and the Y-axis is the values of each parameter. We can easily find the approximate pixel positions of specular area (100–170) from Figure 5b since specular has much higher grayscale values. So in Figure 5b, we can obtain the threshold of $Dol{p}_r\left(x\right)$ is 0.1; in Figure 5c, the threshold of $Dol{p}_g\left(x\right)$ is 0.2; in Figure 5d, the threshold of $Dol{p}_b\left(x\right)$ is 0.25. For the luminance $l\left(x\right)$, we can see the threshold is about 130 in Figure 5e, so the final threshold of $l\left(x\right)$ is 0.5 after normalization.In a conclusion, we set $V_0=\left[0.1,0.2,0.25,0.5\right]$. The value of $\epsilon =0.3$ is obtained by comparing the experimental results, and the best result of specular highlight area detection can be obtained when $\epsilon =0.03$. The specular highlight detection result is shown in Figure 6.

3.2.2. Highlight Removal Based on Multiband Polarization

Compared with the unpolarized diffuse reflection component, the specular highlight component has different polarization states in different wavebands. Therefore, in this section, a Max-Min multi-band-polarization differencing scheme is proposed to generate a single specular-free image (SSF) by utilizing the multi-band polarization characteristic differences between diffuse reflection component and specular highlight component. Meanwhile, for the detected highlight area acquired in the previous section, an ergodic least-squares coefficients decomposition strategy is proposed to obtain the reflection coefficients of diffuse reflection and highlight and thus the effective separation of specular highlight can be achieved.

a.

Max-Min multi-band-polarization differencing scheme

Combining multi-band-polarization imaging model (7) and dichromatic reflection model (2), we have:

$$I_{\lambda }=I_{\lambda ,d}+f·I_{\lambda ,s}={\rho }_d{D}_{\lambda }+{\rho }_s{S}_{\lambda }$$

(15)

It is assumed that the light source has a uniform energy distribution in the visible bands, so for $R,G,B$ three visible wavebands, we have:

$$S=[S_R,S_G,S_B]=[255,255,255]$$

(16)

For the visible wavebands $\lambda \in \left\{R,G,B\right\}$, we can define the Max-Min multi-band-polarization differencing image $\mathsf{\Delta }I$ as:

$$\mathsf{\Delta }I=max{I}_{\lambda }-min{I}_{\lambda }$$

(17)

Combining the three equations above, we have:

$$\begin{array}{cc}\mathsf{\Delta }I& =I_{max}-I_{min}\\ & ={\rho }_d{D}_{max}+{\rho }_{s}S-({\rho }_d{D}_{min}+{\rho }_{s}S)\\ & ={\rho }_d(D_{max}-D_{min})\end{array}$$

(18)

It can be derived that the Max-Min multi-band-polarization differencing image is only related to the diffuse reflection component rather than the specular highlight. Therefore, spectral differential images in three polarization angles—0°, 45°, and 90°—are merged with the linear weighting to generate a single specular-free (SSF) image, which can be expressed as:

$$SSF=\frac{1}{M}\sum _{\phi =0^{\circ },{45}^{\circ },{90}^{\circ }}m\left(\phi \right)\mathsf{\Delta }I\left(\phi \right)$$

(19)

where $\phi$ represents rotation angle of the polarizer, and $m(·)$ denotes the adaptive weight coefficient, which is associated with the intensities of polarization images with different angles, $M={\sum }_{\phi =0^{\circ },{45}^{\circ },{90}^{\circ }}m\left(\phi \right)·\mathsf{\Delta }I\left(\phi \right)$ represents the normalization factor, namely the weighting sum.

As shown in Figure 3, the SSF image can effectively avoid the influence of specular highlight, and meanwhile retaining the original information of the target, which can effectively guide the subsequent removal of the specular highlight.

b.

Ergodic least-squares separation algorithm

For the pixels in the detected highlight area in Figure 6, we can find a diffuse pixel with the closest intensity value in the SSF image, and the intensity of the diffuse pixel is taken as the diffuse intensity of the highlighted pixel. Based on this idea, we propose an ergodic least-squares separation algorithm to perform a global ergodic search in the SSF image, to realize effective guidance of reflection coefficients separation in the detected specular highlight area. The process is as follows:

Step 1: Highlight separation of pixels in highlight area.

For pixel p marked $mask=1$ in specular highlight area, searching for a diffuse reflection pixel q which has the nearest intensity value to p in SSF image, so we have:

$$q^{*}=\underset{q}{argmin}{\left|\right|SSF\left(p\right)-SSF\left(q\right)\left|\right|}_2$$

(20)

Take the original intensity value of q as the diffuse intensity of the highlighted pixel p, and combining with (1), we have:

$$I\left(p\right)={\rho }_d\left(p\right)I\left(q\right)+{\rho }_s\left(p\right)S$$

(21)

Using the least-squares coefficient decomposition method, the reflection coefficient matrix is:

$$\left[\begin{array}{c}{\rho }_d\left(p\right)\\ {\rho }_s\left(p\right)\end{array}\right]={\left[\begin{array}{cc}I\left(q\right)& S\end{array}\right]}^{-1}I\left(p\right)$$

(22)

where “$-1$” represents the pseudo-inverse of the matrix. For the diffuse reflection factor ${\rho }_d\left(p\right)$, its corresponding diffuse reflection intensity can be expressed as:

$$I_d\left(p\right)={\rho }_d\left(p\right)D\left(p\right)={\rho }_d\left(p\right)I\left(q\right)$$

(23)

Furthermore, the intensity of corresponding highlight component is:

$$I_s\left(p\right)=I\left(p\right)-{\rho }_d\left(p\right)I\left(q\right)$$

(24)

After separation, pixel p is marked as a diffuse pixel, that is $mask\left(p\right)=0$. The algorithm then proceeds to the next highlight pixel.

Step 2: Highlight separation of pixels in diffuse reflection area.

For the diffuse reflection area where $mask=0$, some diffuse reflection pixels may contain a small amount of highlight component. To further realize the separation and removal of specular highlight, we use the same least-squares strategy to obtain the diffuse reflection and specular highlight coefficients.

For the diffuse reflection area where $mask=0$, searching for a diffuse reflection pixel r whose intensity value is the highest in the SSF image, namely:

$$r^{*}=\underset{r}{argmax}SSF\left(r\right)s.t.mask\left(r\right)=0$$

(25)

Then seeking a diffuse reflection pixel z with the nearest intensity value to pixel r in the SSF image, which satisfies:

$$z^{*}=\underset{z}{argmin}{\left|\right|SSF\left(r\right)-SSF\left(z\right)\left|\right|}_2$$

(26)

Next taking the original intensity value of r as diffuse reflection intensity of pixel z, the reflection coefficient matrix can be calculated by using the least-squares coefficient decomposition method:

$$\left[\begin{array}{c}{\rho }_d\left(z\right)\\ {\rho }_s\left(z\right)\end{array}\right]={\left[\begin{array}{cc}I\left(r\right)& S\end{array}\right]}^{-1}I\left(z\right)$$

(27)

Since the reflection coefficient of highlight is non-negative, if the calculated ${\rho }_s\left(z\right)<0$, (24) can be converted to:

$${\rho }_d\left(z\right)=I{\left(r\right)}^{-1}I\left(z\right)$$

(28)

Similar to (20) and (21), its corresponding diffuse reflection and specular highlight components can be calculated. Finally, pixel z is marked as a processed pixel, and then the same procedure can be adopted to the next diffuse reflection pixel which has the nearest intensity value to pixel r according to (23).

In summary, our proposed method considers not only the reflection separation of pixels in the highlight area, but also the possibility of the existence of a small amount of highlight in the diffuse reflection area, so that the removal result is much more thorough and accurate.

c.

The compensation of missing information based on local chromaticity consistency regularization constraint

The proposed highlight removal algorithm in Section $\mathit{b}$ ignores the local detail information of the original target and leads to partial color distortion and edge discontinuity after specular highlight removal. So we use our previous work, local chromaticity consistency [46], to realize weighting regularized constraint of highlight removal result and meanwhile variable splitting [47] is combined to achieve fast optimization solving. In this way, the missing information after highlight removal can be effectively compensated in time and the visual effect is further improved. For the highlight suppression result $I_d$, the weighting regularization term can be denoted as $W(x,y)|I_d\left(x\right)-I_d\left(y\right)|$. The pixels with a similar grey level neighborhood to $I_d$ have larger weights in the average, thus like that in nonlocal means methods [48], $W(x,y)$ can be set to be inversely proportional to the distance between pixel x and y and can be expressed as:

$$W(x,y)=e^{-{\left|\right|I\left(x\right)-I\left(y\right)\left|\right|}^2/2{\sigma }^2}$$

(29)

where I is the original target image, and $\sigma$ represents standard deviation. Therefore, for two adjacent pixels x and y in a local area, when this two pixels belong to diffuse and highlight areas, respectively, significant difference exists between $I\left(x\right)$ and $I\left(y\right)$, namely $W(x,y)\approx 0$; while if this two pixels belong to the same reflection area, the weighted value $W(x,y)$ is bigger.

To simplify, the matrix forms of the weighting regularization term and weighting function (26) can be expressed as $\left|\right|W\circ (D\otimes I_d){\left|\right|}_1$ and $W=e^{-{\sum }_{i\in r,g,b}{\left|D\otimes I_i\right|}^2/2{\sigma }^2}$, where D is first-order forward difference operator, ∘ represents matrix product, ⊗ represents convolution and i is color channel. Therefore, the problem of weighting regularization information compensation can be converted into the following energy function minimization problem:

$$\underset{I_d}{argmin}\frac{\gamma }{2}{∥I_d-\widetilde{I_d}∥}_2^2+{∥W\circ (D\otimes I_d)∥}_1$$

(30)

where $\gamma$ is weighting factor, $\widetilde{I_d}$ is the initial highlight removal result. For the above energy function optimization problem, we can quickly solve it by the variable splitting method. Here we introduce an intermediate variable u, so (27) can be converted into:

$$\frac{\gamma }{2}{∥I_d-\widetilde{I_d}∥}_2^2+{∥W\circ u∥}_1+\frac{\beta }{2}\left({∥u-(D\otimes I_d)∥}_2^2\right)$$

(31)

where $\beta$ is a weighting factor. Obviously, (28) converges to the optimal solution of (27) when $\beta \to \infty$.

For (28), when $\beta$ is fixed, the minimization problem can be treated as alternating optimization solution of u and $I_d$. That is, first fixing $I_d$ to calculate optimal u; then fixing u to obtain optimal $I_d$. The process continues until convergence. The detailed optimization process is as follows:

(1)

Fixing $I_d$ and optimizing u

In (28), for a given $I_d$, its minimum energy function is:

$${∥W\circ u∥}_1+\frac{\beta }{2}{∥u-D\otimes I_d∥}_2^2$$

(32)

It can be converted into:

$$\underset{x}{min}\left|\omega ·x\right|+\frac{\beta }{2}{\left(x-\alpha \right)}^{{}^2}$$

(33)

where are $\omega$, $\alpha$, $\beta$ all known and can be obtained directly, namely:

$$u^{*}=x_{o}pt=max((\left|\alpha \right|-\frac{\omega }{\beta },0)·sign\left(a\right)$$

(34)

here $sign(·)$ is the symbol function.

(2)

Fixing u and optimizing $I_d$

We regard the calculated u as a fixed value to obtain the optimal $I_d$. The objective function is:

$$\frac{\gamma }{2}{∥I_d-\widetilde{I_d}∥}_2^2+\frac{\beta }{2}\left({∥u-(D\otimes I_d)∥}_2^2\right)$$

(35)

Notice that (35) is a quadratic term function of the variable $I_d$, so we can obtain the optimal value of $I_d$ by taking the partial derivative of (35) and we have:

$$\frac{\gamma }{\beta }(I_d-\widetilde{I_d})+D^T\otimes (D\otimes I_d-u)$$

(36)

where $D^T$ is the transpose matrix of D. Then we set (36) to be zero to obtain the optimal value of $I_d$:

$$\frac{\gamma }{\beta }(I_d-\widetilde{I_d})+D^T\otimes (D\otimes I_d-u)=0$$

(37)

$$\frac{\gamma }{\beta }{I}_d+D^T\otimes D\otimes I_d=D^T\otimes u+\frac{\gamma }{\beta }\widetilde{I_d}$$

(38)

Since convolution calculation is complicated in the time domain, so we use 2D FFT (Fast Fourier Transform) to calculate the optimal value of $I_d$ in the frequency domain. Thus, we have:

$$\frac{\gamma }{\beta }\circ F\left(I_d\right)+\overline{F\left(D\right)}\circ F\left(D\right)\circ F\left(I_d\right)=\overline{F\left(D\right)}\circ F\left(u\right)+\frac{\gamma }{\beta }F\left(\widetilde{I_d}\right)$$

(39)

$$F\left(I_d\right)=\frac{\frac{\gamma }{\beta }F\left(\widetilde{I_d}\right)+\overline{F\left(D\right)}\circ F\left(u\right))}{\frac{\gamma }{\beta }+\overline{F\left(D\right)}\circ F\left(D\right)}$$

(40)

where $F(·)$ represents FFT and $\overline{(·)}$ denotes complex conjugate. Finally the optimal value of $I_d$, denoted as $I_d^{*}$, can be obtained by Inverse Fast Fourier Transform (IFFT) transformation:

$$I_d^{*}=F^{-1}\left(\frac{\frac{\gamma }{\beta }F\left(\widetilde{I_d}\right)+\overline{F\left(D\right)}\circ F\left(u\right))}{\frac{\gamma }{\beta }+\overline{F\left(D\right)}\circ F\left(D\right)}\right)$$

(41)

where $F^{-1}(·)$ represents IFFT.

4. Experiments and Results

In order to verify the effectiveness and wide applicability of our proposed specular highlight removal algorithm, we conducted experiments on different fruits with smooth skin and less texture. And six existing algorithms, including Mallick [18], Shen [19], Shen [21], Akashi [23], Yamamoto [49] and Fu [26], are selected to be compared with our proposed algorithm. Since the fruit target is delivered to the image acquisition room by conveyor belt in the automatic quality inspection system as shown in Figure 1, we cannot be completely sure about the position and quantity of the fruit target. Therefore, the distribution of highlights on the fruit’s surface is naturally different and multiple fruits may be captured at the same time in one image acquisition. So here we selected three representative fruit targets with different quantities and highlight distributions, the first target is one apple with the concentrated highlight, and the second is one lemon with dispersive highlight distribution and the third target is three oranges with dispersed highlight.

4.1. Experimental Data Acquisition

Since the imager we designed in Section 3.1 is still in the theoretical research stage, so we use the LUCID PHX0505 camera to acquire the images of targets and the size of the captured images is 2448 × 2048. In total, we captured images of 50 different scenes. Furthermore, as mentioned before, three groups of representative experimental results are presented in the manuscript. Due to the nature of spatial sampling, the image captured by our designed imager is a mosaic image containing all the required information in three wavebands and three polarization angles. The mosaic images of our selected three representative fruit targets are shown in Figure 7(a1,b1,c1). Demosaicing method [11] is used here to reconstruct color images at three polarization angles—0°, 45°, and 90°. The reconstruction results are shown in Figure 7(a2–a4) in the first row shows the reconstruction results of an apple. Similarly, the reconstruction results of a lemon and three oranges are shown in Figure 7(b2–b4) and Figure 7(c2–c4), respectively.

4.2. Objective Evaluation Results

To objectively evaluate the quality of highlight removal results, three objective evaluation indexes: average gradient ($AG$), angular second moment ($ASM$) and inverse difference moment ($IDM$) [50] are chosen to quantitatively analyze the specular highlight removal results.

The $AG$ calculates the gradient amplitude in the horizontal and vertical directions of the image, which further shows the detail contrast and texture change characteristics in the highlight removal results. Furthermore, a larger value of $AG$ means richer image detail information. The expression of $AG$ is:

$$AG=\frac{1}{(M-1)(N-1)}\sum _{i=1}^{M-1}\sum _{j=1}^{N-1}\sqrt{\frac{{\left(\frac{\partial I(i,j)}{\partial i}\right)}^2+(\frac{\partial I(i,j)}{\partial j}{)}^2}{2}}$$

(42)

The $ASM$ is the sum of squares of the elements in the gray co-occurrence matrix, which indicates the distribution uniformity of gray value and the degree of texture fineness of the highlight removal results. Therefore, a larger value of $ASM$ means richer image texture information and more uniform local distribution, and that means the highlight removal effect is better. The calculation formula of $ASM$ is:

$$ASM=\sum _i^{}\sum _j^{}P{(i,j)}^2$$

(43)

$IDM$ represents the regularity of texture in highlight removal results. Furthermore, a smaller value of $IDM$ means richer image texture, more complex structure and higher detail information retention. The expression of $IDM$ is:

$$IDM=\sum _i^{}\sum _j^{}\frac{1}{1+{(i-j)}^2}P(i,j)$$

(44)

The objective evaluation results of the three scenarios are shown in

Table 1. The Quantitative Evaluation of Highlight Removal Results of 1st Scenario.

	Mallick [18]	Shen [19]	Shen [21]	Akashi [23]	Yamamoto [49]	Fu [26]	Proposed
AG	0.2792	0.2622	0.2047	0.2282	0.2457	0.2523	0.3101
ASM	0.6290	0.6001	0.5986	0.6131	0.5869	0.6020	0.6335
IDM	0.9953	0.9969	0.9973	0.9958	0.9989	0.9965	0.9941

,

Table 2. The Quantitative Evaluation of Highlight Removal Results of 2nd Scenario.

	Mallick [18]	Shen [19]	Shen [21]	Akashi [23]	Yamamoto [49]	Fu [26]	Proposed
AG	0.2468	0.2515	0.2499	0.2582	0.2601	0.2597	0.2826
ASM	0.7002	0.7065	0.7021	0.7030	0.7055	0.7064	0.7093
IDM	0.9971	0.9969	0.9963	0.9958	0.9957	0.9956	0.9948

and

Table 3. The Quantitative Evaluation of Highlight Removal Results of 3rd Scenario.

	Mallick [18]	Shen [19]	Shen [21]	Akashi [23]	Yamamoto [49]	Fu [26]	Proposed
AG	0.2452	0.2285	0.2447	0.2382	0.2449	0.2503	0.2714
ASM	0.7164	0.7065	0.7119	0.7131	0.7187	0.7174	0.7182
IDM	0.9959	0.9979	0.9964	0.9969	0.9966	0.9957	0.9953

, respectively. The best results are shown in bold. The average values and standard deviation of quantitative evaluation results are listed in

Table 4. Average Values and Standard Deviation of Quantitative Evaluation (Average Values/Standard Deviation).

	Mallick [18]	Shen [19]	Shen [21]	Akashi [23]	Yamamoto [49]	Fu [26]	Proposed
AG	0.2605/0.0249	0.2487/0.0222	0.2235/0.0209	0.2381/0.0245	0.2488/0.0267	0.2583/0.0278	0.2921/0.0194
ASM	0.6196/0.0373	0.6725/0.0368	0.6085/0.0357	6780/0.0341	0.6120/0.0387	0.6753/0.0402	0.6855/0.0358
IDM	0.9951/0.0010	0.9973/0.0012	0.9964/0.0009	0.9967/0.0010	0.9950/0.0012	0.9956/0.0014	0.9945/0.0009

.

4.3. Specular Highlight Removal Results

We conduct experiments on three scenarios to verify the effectiveness of our proposed method. Furthermore, here six existing algorithms, including Mallick [18], Shen [19], Shen [21], Akashi [23], Yamamoto [49] and Fu [26], are selected to be compared with our algorithm for visual evaluation. It should be noted here that for these compared algorithms, the 0° polarization image is selected as the input image for subsequent highlight removal. The specular highlight removal results are shown in Figure 8, Figure 9 and Figure 10.

4.4. Quality Inspection Results

To further verify the effectiveness of our proposed specular highlight removal algorithm, we conduct fruit quality inspection based on the specular highlight removal results obtained in Section 4.3. Firstly, we choose a relatively simple method based on edge detection [51] for fruit damage detection. Furthermore, the detection results of three targets are shown in Figure 11, Figure 12 and Figure 13.

Secondly, the fruit quality inspection can be realized by calculating the proportion (%) of the damaged area to the total image area [52]. Furthermore, the standard is: If the proportion of the damaged area to the total image area obtained above is less than $2\%$, the fruit target is considered “Good”, while if the proportion is larger than $2\%$, the fruit target is considered “Damage”. The quality inspection results are shown in

Table 5. The Quality Inspection Results of 1st Scenario.

	Mallick [18]	Shen [19]	Shen [21]	Akashi [23]	Yamamoto [49]	Fu [26]	Proposed
Results	Damage	Damage	Damage	Damage	Damage	Damage	Good

,

Table 6. The Quality Inspection Results of 2nd Scenario.

	Mallick [18]	Shen [19]	Shen [21]	Akashi [23]	Yamamoto [49]	Fu [26]	Proposed
Results	Damage	Damage	Damage	Good	Good	Damage	Good

and

Table 7. The Quality Inspection Results of 3rd Scenario.

	Mallick [18]	Shen [19]	Shen [21]	Akashi [23]	Yamamoto [49]	Fu [26]	Proposed
Results	Damage	Damage	Damage	Damage	Damage	Damage	Good

.

5. Discussion

In this study, we propose a specular highlight removal method for quality inspection of fruits. The effectiveness of the proposed method is verified through (1) objective evaluation indexes (AG, ASM, IDM), (2) visual effects and (3) fruit quality inspection results. A detailed discussion of the obtained experimental results will be presented in this section.

The quantitative evaluation results are shown in Table 1, Table 2 and Table 3. The three objective evaluation indexes are mainly used to judge the preservation degree of texture details, so in Table 1, the results of Shen [21] are the worst since serious data loss exists after the removal of the highlight. The results of other comparing methods are not ideal since they all fail to remove the specular highlight completely. Our proposed method separates the highlight more accurately and then performs information compensation, so our results are the best. The results of Mallick are worst in Table 2 due to effective color space conversion cannot be conducted for the specular highlight areas. The results of Shen [19] are the worst in Table 3, which is also due to the information loss caused by specular highlight removal. In a conclusion, our proposed algorithm outperforms the other six comparing algorithms and the objective evaluation results are all the best in Table 1, Table 2 and Table 3.

For the first fruit target (an apple), the reconstructed polarization images, and highlight removal results of our proposed algorithm and six other comparing algorithms are shown in Figure 8. It can be seen clearly that Shen’s [19] method has serious information loss after removing the specular highlight. Furthermore, the removal results of Mallick, Shen [21], Akashi, Yamamoto [49] and Fu have different degrees of highlight residue. Besides, a small amount of data is lost in Akashi’s Yamamoto’s and Fu’s results. In summary, the comparing six methods cannot effectively remove the specular highlight, and serious color distortion or data void occurs. In contrast, our proposed method has a much better highlight removal effect and higher reliability. The reason is that the proposed algorithm is based on the constraints of physical feature analysis, and the missing details are compensated based on the spatial color consistency regularization after the specular highlight removal. As shown in Figure 8(b7), our removal result has clearer texture on the target surface and better color fidelity, meanwhile the edge detail information is greatly preserved. The visual effect of our method is better than that of the other algorithms.

For the 2nd scene (a lemon), the specular highlight has dispersed distribution. The highlight removal results are shown in Figure 9. We can see that highlight residue exists in both Mallick’s and Shen’s [21] results as shown in Figure 9(b1,b3), respectively. The specular highlights are completely removed in the results of Shen [19], Akashi, Yamamoto [49] and Fu, but it can be seen clearly that data loss degrade the image qualities. However, the input of our algorithm is multidimensional information including intensity, polarization and spectral information, and the richness of the input information makes the proposed method has higher accuracy in specular highlight detection, removal and subsequent compensation of lost information, so our proposed method has a relatively better visual effect. Although there still exists very little specular highlight, the original color, texture and other detailed information of the target surface is retained well.

For the 3rd scene (oranges), its surface is a biological greasy epidermis with single surface color and simple texture, and the specular highlight area is large and dispersed in the target image. The highlight removal results are shown in Figure 10. Since the color space conversion cannot be effectively performed in the specular highlight area, it can be seen that in Mallick’s results, the highlight cannot be accurately separated and meanwhile data voids exist. The result of Shen [19] contains a large area of data voids. Furthermore, the same problems also exist in the results of Shen [21], Akashi [23], Yamamoto [49] and Fu [26]: blurred and distorted edges, and data loss. We can see that in the results of the first six methods, most of the information in the highlighted area is lost, and a black circle appears in each removal result, which will certainly affect the subsequent quality inspection results. In comparison, our proposed method has the best visual effect as shown in Figure 10(b7). Similarly, the combination of multidimensional information input and compensation of missing details guarantees that our method has better visual performance. It can be seen that the specular highlight is removed thoroughly, and meanwhile the original color, texture and other detailed information of the target surface is retained efficiently.

Figure 11, Figure 12 and Figure 13 show the damage detection results of three scenarios. It can be seen that the highlight removal results of other comparing algorithms are detected as having damage since they fail to remove the specular highlight accurately and completely. Wrong damage detection results lead to subsequent erroneous quality inspection results. By contrast, our damage detection results shown in Figure 11(7), Figure 12(7) and Figure 13(7) are accurate, and the quality inspection results are correct as shown in Table 5, Table 6 and Table 7 as well. This further illustrates the effectiveness of our proposed specular highlight removal algorithm, and it also proved that our proposed method is of great importance to improve the accuracy of fruit detection.

The running time of the comparing algorithms and our proposed method are shown in

Table 8. Running Time of Each Algorithm (Unit: seconds).

	Mallick [18]	Shen [19]	Shen [21]	Akashi [23]	Yamamoto [49]	Fu [26]	Proposed
Running time	21.31 s	5.74 s	1.18 s	81.14 s	94.57 s	345.03 s	55.67 s

. Our proposed method is not the fastest. There are three reasons for the large gap in running time between Shen’s two methods, Mallick’s method and ours. Firstly, the computation of our proposed algorithm is relatively large. Since three procedures are included in our algorithm: the detection of specular highlight based on the joint multi-band-polarization characteristic vector constraint; the removal of specular highlight based on the proposed Max-Min multi-band-polarization differencing scheme combined with an ergodic least-squares separation; and the compensation of missing details based on the chromaticity consistency regularization. Secondly, our acquired image in real-time is a mosaic image containing all the required information in three wavebands and three polarization angles, so the demosaicing method is required to realize information reconstruction. The demosaicing process takes some time. Last but not least, the input image becomes multiple color polarization images after demosaicing, so there is no doubt that the processing time must be longer than the first three methods since they are all single-image-input methods, so our time is definitely longer. However, the accuracy of our algorithm is much higher than these three algorithms. Our algorithm code is based on the MatLab platform, and we can accelerate it by moving it to a new platform such as FPGA in the future.

6. Conclusions

In this paper, we propose a specular highlight removal algorithm based on multi-band polarization imaging for quality inspection of fresh fruits. In the proposed algorithm, we first design a new multi-band polarization imager to realize real-time image acquisition. Then, we propose a specular highlight removal method to accomplish high-precision highlight removal. Experimental results demonstrate that the proposed method is highly efficient and robust; meanwhile, it can effectively remove the specular highlight of fruits with smooth skin and less texture. Both the visual effect, image quality and objective evaluation indices are better than existing algorithms. In addition, our proposed algorithm can be directly integrated into existing quality inspection equipment, which is of great importance to improve the performance of existing quality inspection equipment.