Sandstorm Image Enhancement Using Image-Adaptive Eigenvalue and Brightness-Adaptive Dark Channel Network

Abstract

Sandstorm images suffer from dust particles and the attenuation of light. Degraded sandstorm images have a reddish or yellowish color cast owing to dust particles, which have a certain color. Because of this phenomenon, distorted sandstorm images have imbalanced red and blue channels. Existing sandstorm enhancement methods improve the image by only focusing on dehazing, due to sandstorm images having similar features to hazy or dusty images. However, the enhanced images using previous methods have a color cast. Therefore, to enhance the sandstorm image naturally, a balancing procedure of color components is needed. This paper proposes a color-balancing method using image-adaptive eigenvalues. Because eigenvalues describe image characteristics, improved images have balanced color components. However, these enhanced images have dusty features. Therefore, to enhance balanced images a dehazing procedure is needed, something which this paper applies using a multiscale convolution neural network, thus generating a transmission map with brightness-adaptive features. Images enhanced using the proposed method compare well with state-of-the-art methods in subjective and objective measures of quality.

1. Introduction

The image under sandstorm circumstances has color-distorted features because of the imbalanced color channels, in turn due to attenuation. This results in issues in computer vision, robot vision, and image recognition. The color channels of degraded sandstorm images exhibit asymmetry, whereby the red channel is abundant, and the blue channel is sparse due to attenuation and scattering. If the particles have no color components, the sandstorm image has only dusty or hazy features. However, if the particles have color, then the obtained image has a certain reddish or yellowish color cast. If a color-balancing procedure is not considered, then the enhanced image has an artificial color. Therefore, enhancement of the degraded image is needed.

Improvement methods of sandstorm images can be classified into model-based methods [1,2,3,4,5], model-free based methods, and machine learning-based methods.

Because sandstorm images and hazy images are obtained similarly, model-based [1,2,3,4,5] algorithms have been studied to enhance them. He et al. improved hazy images using the dark channel prior to estimate the darkest region in the image [5]. However, the enhanced image had artificial color components due to this method [5] being unable to sufficiently estimate the dark region in bright images. Meng et al. [6] enhanced images using a refined transmission map. This method [6] improved the image using a boundary constraint transmission map, which better estimated the transmission map compared to the method that of He et al. [5]. However, this method also had a weak point in bright regions. Lee’s [7] method improved sandstorm images using a normalized eigenvalue and the brightness-adaptive dark channel prior method. This method [7] enhanced degraded sandstorm images in spite of color distortion. Gao et al. [8] improved degraded sandstorm images using the reversed blue channel prior method, considering its sparsity in distorted sandstorm images. This method [8] led to less of a color shift; however, a hazy effect remained. Shi et al. [9] enhanced the sandstorm image using the mean shift of color components and an image-adaptive transmission map. However, this method [9] led to an artificial color shift in some images due to the shift of color components, regardless of color channel abundance. Wang et al. enhanced the hazy image using linear transformation [10]. This method estimates the atmospheric light based on quad tree subdivision, and its weakness is that this method displays a darker image when the dehazing level is increased [10].

Model-free based sandstorm image enhancement algorithms use image processing techniques such as histogram equalization and gamma correction. Al Ameen [11] enhanced sandstorm images using gamma correction. This method could enhance lightly degraded sandstorm images. However, in the case of greatly distorted images, the enhanced images had artificial color components because of the constant value used to operate gamma correction. Besides, a constant gamma value is not able to sufficiently reflect the image’s characteristics. Shi et al. proposed a sandstorm image enhancement method using the mean shift of color components and CLAHE [12]. This method could enhance the degraded sandstorm images; however, in the case of images with a sparse blue channel, the enhanced images had artificial color components. Cheng et al. improved sandstorm images using a robust gray world assumption and guided image filtering [13]. Zhu et al. improved the hazy image using gamma correction and multiexposure image fusing [14]. This method enhances the hazy image well. However, in some images a color shift effect appears.

Sometimes, the denoising algorithm is adopted in the dehazing area partially [15,16] due to the haze being also a particle of something. Mahdaoui et al. enhanced the noisy image using augmented Lagrangian methods [15].

Recently, machine learning-based sandstorm enhancement algorithms have been studied. Ren et al. [17] enhanced hazy images using a multiscale convolution neural network (CNN). Li et al. [18] improved hazy images using recombination of a hazy model to generate an enhanced haze-free image. Santra et al. successfully enhanced hazy images using a multiscale CNN and environmental illustration corresponding to the input image and its transmission map [19]. However, in some cases such as nighttime images, the enhanced images had an artificial effect due to synthetic images being made from daytime images [19]. Wang et al. enhanced hazy images using the atmospheric illumination prior method [20]. This method used the luminance channel to enhance hazy images with a multiscale CNN. Zhang et al. also improved hazy images using a multiscale CNN featuring encoders with three scaled convolutional layers and pooling layers constituting a fusion module [21].

Machine learning-based methods have advantages in some image processing areas. However, machine learning-based methods, because of the limited and insufficient dataset conditions, have disadvantages in naturally and sufficiently enhancing images in all conditions. Meanwhile, if the synthetic dataset is generated by reflecting the natural conditions and trained, then good performance is achieved even though the original dataset is limited. Sandstorm images have degraded color components owing to scattered, attenuated, and imbalanced color channels. Therefore, to naturally enhance distorted sandstorm images, it is necessary to balance the color channels. This paper proposes a color-balancing method using image-adaptive eigenvalues. The eigenvalues of an image reflect its features [7]. Accordingly, degraded color channels can be balanced to naturally enhance the image. Because the balanced images have similar features to hazy or dusty images, a further dehazing procedure is needed. Existing dehazing methods frequently use the DCP [5]. However, this method struggles in bright regions and constant kernel size, with the enhanced image having artificial color components in some cases because of the improperly estimated DCP. Therefore, this paper enhances hazy images using the DCP, based on a multiscale CNN composed of various layers. Accordingly, there is no need for a constant block region, and the enhanced image has no artificial effect.

Images enhanced using the proposed method compare well with state-of-the-art methods in subjective and objective measures of quality.

2. Proposed Method

Sandstorm images and dusty images are obtained similarly. However, sandstorm images have shifted color components due to colorful sand particles. The characteristics of sandstorm images are various due to attenuation or scattering by sand particles. Figure 1 shows the sandstorm images. Figure 1a,b show a nondegraded dusty image and its histogram distribution of color channel, while Figure 1c,d show a degraded sandstorm image and histogram distribution of color channel. As can be seen, in order to balance the color-casted sandstorm image, the color channels of the image require balancing.

Therefore, this section proposes a method for balancing color channels using image-adaptive eigenvalues.

2.1. Image-Adaptive Color Balance

When light propagates through the sand/dust medium, scattering and attenuation render the obtained image with a reddish or yellowish color distortion. Therefore, to naturally enhance color-casted sandstorm images, a color-balancing procedure is needed. Lee [7] enhanced degraded color channels using eigenvalues. This method performed well in enhancing distorted sandstorm images. However, this method uses the normalized eigenvalue of each color channel with respect to the maximum value, which sometimes leads to an overflow region due to the asymmetric distribution of red and blue channels. Therefore, this paper proposes image-adaptive eigenvalues to balance degraded color channels.

The eigenvalue of an image is obtained by [7,22,23]:

$$I·V=\lambda ·V$$

(1)

where $I$ is image matrix, $V$ is non-zero vector, and $\lambda$ is eigenvector. Equation (1) is able to be reformed as:

$$I·V-\lambda ·V·U=0$$

(2)

where $U$ is unit matrix. Because the $V$ is non-zero vector, the eigenvalue of image is obtained by

$$\det \left(I-\lambda ·U\right)=0$$

(3)

where $\det (·)$ is determinant operation. To acquire the eigenvalue of an image needs a square matrix. However, the natural image has diverse image sizes. Therefore, in order to fit the same image size on row and col, the size of image is fit in min between row and col. The characteristic of an eigenvalue according to color channel is shown in the Lee method [7]. If the image has color cast as with being reddish, then the eigenvalue of red channel is higher than that of green and blue. Additionally, using this property, the color-casted image is able to be balanced. The cue point of the Lee method [7] to balance the color is described as [7]:

$${\mathsf{\lambda }}_n^c=\frac{{\mathsf{\lambda }}_{max}^c}{\underset{c}{\max }({\mathsf{\lambda }}_{max}^c)}$$

(4)

where ${\mathsf{\lambda }}_n^c$ is the normalized eigenvalue of each color channel with respect to the maximum value, ${\mathsf{\lambda }}_{max}^c$ is the maximum eigenvalue of each color channel, and $c\in \left\{r,g,b\right\}$. This method performs nicely to balance the color-casted image. However, this method [7] balances the image just using normalized eigenvalue among greatly attenuated channel and abundant channel and because of this point the balanced image has an overflowed region. Therefore, this paper proposes the image-adaptive color-balancing method by reflecting the characteristics of the color channel. The balancing procedure can be expressed as follows:

$$I_b^c\left(x\right)=I^c\left(x\right)·{\mathsf{\lambda }}_{\alpha }{}^c+{\beta }^c·\left(1-I^c\left(x\right)\right)·{\mu }_{I^c\left(x\right)}$$

(5)

$${\mathsf{\lambda }}_{\alpha }^c={\log }_2\left(1+\frac{{\mu }_{{\mathsf{\lambda }}_n^{}}}{{\mathsf{\lambda }}_n^c}\right)$$

(6)

$${\beta }^c=\mu \left({\mu }_{I^c\left(x\right)}\right)-{\mu }_{I^c}$$

(7)

where $\mu$ is the average operation. In Equation (6), ${\mathsf{\lambda }}_n^c$ was proposed by Lee [7] as a function with the maximum eigenvalue of each color channel to balance the degraded sandstorm image. However, this method leads to the above mentioned drawbacks. Therefore, to compensate for this phenomenon, this paper applies the image-adaptive eigenvalue ${\mathsf{\lambda }}_{\alpha }^c$ using Equation (6). By multiplying the average value of ${\mathsf{\lambda }}_n^c,$ the excess value can be suppressed, leading to a more naturally balanced image. Moreover, using Equation (5), sparse color components can be compensated for. Equation (5) is somewhat similar to that proposed by Ancuti et al. [24]. However, Ancuti et al.’s [24] method enhanced images by compensating for sparse color components using the green channel. However, the proposed method used the gray channel consisting of the average value of all image features to reflect the channel conditions. Using this method, the relatively abundant red channel can be reduced, while sparser components can be compensated for. ${\beta }^c$ controls the image balance using grayscale if the red channel is abundant. The balanced image obtained using Equations (5)–(7) has no color cast and appears natural.

Figure 2 compares the balanced images obtained using Lee’s method [7] and the proposed method. As shown in Figure 2, the image enhanced using the proposed method has no color shift or overflow region, as well as a uniformly distributed histogram. Therefore, the proposed color-balancing method is promising for future applications.

2.2. Dehazing Using Brightness-Adaptive Dark Channel Network

Images balanced image using the proposed method seem hazy or dusty without a color shift. Therefore, a dehazing procedure is proposed. The dark channel prior (DCP) method [5] is frequently used to enhance the hazy image. However, this method [5] has disadvantages when estimating bright regions, resulting in an artificial color shift in the enhanced image. Therefore, to compensate for this effect, Lee proposed a brightness-adaptive transmission map using reverse DCP [7]. However, because the dark channel approach consists of a constant block region, the estimated image has a block effect. Moreover, He et al. [5] used a refinement procedure with a guided image filter [25] and a bilateral filter [26,27] to enhance the artificial effect of DCP. Therefore, the DCP method [5] and Lee’s method [7] can be used enhance hazy images; however, an artificial color shift occurs. The existing transmission map procedure uses a constant kernel size to estimate the dark region of each image.

$$I_{DC}\left(x\right)=\underset{c}{\min }(\underset{y\in \mathsf{\Omega }\left(x\right)}{\min }(\frac{I^c\left(y\right)}{A^c}))$$

(8)

where $I_{DC}\left(x\right)$ is the estimated dark channel, $\mathsf{\Omega }\left(x\right)$ is the patch size (which was set to 15 in He et al.’s [5] method), and $A^c$ is the backscattered light of each color channel according to He et al.’s [5] method. As expressed in Equation (8), a constant patch size is used to estimate the image’s dark region. However, because the kernel size is constant, the estimated dark image has a block effect, which persists in the transmission map. To address this phenomenon, various refinement procedures can be applied. However, the CNN is able to generate the image’s transmission map without using a certain constant kernel size.

Therefore, this paper proposes a brightness-adaptive dark channel using multiscale CNN, thereby eliminating the block effect.

The basic dataset used to train a brightness-adaptive dark channel network is the D hazy dataset [28]. This dataset has a clear scene radiance, a synthetic hazy image, and a depth map. Because the dehazing procedure uses a transmission map, a dataset transmission map can be generated from the depth map. The transmission map can be expressed as follows [1,2,3,4,5]:

$$t\left(x\right)={\exp }^{-\beta ·d\left(x\right)}$$

(9)

where $\beta$ is a scattering parameter, and $d\left(x\right)$ is the depth map of the image. The scattering parameter $\beta$ controls the transmission map’s brightness as with an ‘aerial perspective’ [5,29,30]. The ‘aerial perspective’ indicates haze as a function of distance, with greater values indicating thick haze [5,29,30]. A natural image has various depth features. Therefore, by considering the characteristics of the scattering parameter, the trained transmission map contains various depth maps, thus producing a proper transmission map. The variation of transmission maps with the scattering parameter is displayed in Figure 3. This paper uses the transmission map generated via the scattering parameter with a depth map as the target image for training. To generate various target transmission maps, the scattering parameter ($\beta$) was set in the range [0.1, 0.95, …, 0.75, 0.7] with a 0.05 interval.

The network followed a hybrid of DCP theory [5] and brightness-adaptive DCP [7].

$$l_{mp}\left(x\right)\approx \underset{c}{\min }\left(\underset{y\in \mathsf{\Omega }\left(x\right)}{\min }\right(\frac{I^c\left(y\right)}{A^c}\left)\right)$$

(10)

where $A^c$ is backscattered light according to He et al.’s method [5], $\mathsf{\Omega }\left(x\right)$ is the patch region, and $l_{mp}\left(x\right)$ is part of the minimum pooling layer. Equation (10) shows the common point between minimum pooling and DCP [5] in estimating the dark region in an image. In contrast to minimum pooling, the DCP method [5] needs a certain constant patch size, leading to artificial effects in the enhanced image. Therefore, this paper used minimum pooling to properly estimate the image’s dark region.

To design the multiscale networks, U-net [31] was applied. The concept of the proposed transmission network can be expressed as follows [7]:

$$t_p\left(x\right)=ReLU\left(1-\frac{l_d\left(x\right)}{\max \left({\mu }_{l_d},{\mu }_{1-l_d}\right)+\max \left(l_d\left(x\right),1-l_d\left(x\right)\right)}\right)$$

(11)

where $l_d$ is the multiscaled dark channel layer, $\mu$ is the average operation, x is the location of the pixel, $1-l_d$ is the reverse layer, $ReLU(.)$ is the rectified linear unit (ReLU) [32] operation, $\max (.)$ and $t_p\left(x\right)$ are maximum operation and the proposed transmission map respectively.

The network design is depicted in Figure 4. As can be seen, to generate the transmission map with multiple scales, four minimum pooling layers and three upsampling layers are used. To combine the diverse characteristics of features, five concatenation layers are used. The colored arrows in Figure 4 indicate each layer’s operation (convolution, pooling, upsampling, and concatenation). Moreover, the numbers below the block and the width indicate the channel size of the layer. Each convolution layer consists of a ReLU activation function [32]. The transmission map $\left(l_t\right)$ generated after training on the input image has one channel. The blue dotted rectangle indicates the unit layers used in the network. The green dotted rectangle indicates the basic module of the proposed brightness-adaptive dark channel network. As known through Equation (10), minimum pooling and dark channel prior are similar; hence, minimum pooling and the convolution layer were included in the basic module to generate the multiscale dark image. The concatenation layer, was then used to generate the image’s diverse features. The proposed network generates the trained image $l_d\left(x\right)$ by using color-balanced image $I_b\left(x\right)$. Using $l_d\left(x\right)$, the transmission map can be generated as $l_t\left(x\right)$.

2.3. Loss Function

The loss function used was a combination of SSIM loss [33] and MSE loss. When the generated data and ground-truth data are similar, the SSIM loss is larger meanwhile the MSE loss is smaller.

$$Los{s}_p=\phi ·Los{s}_{mse}$$

(12)

$$\phi =\left(1+Los{s}_{ssim}\right)$$

(13)

$$Los{s}_{mse}=\frac{{{\displaystyle \sum }}_{i=1}^{N}e{\left(t_p\left(x_i\right),G\left(x_i\right)\right)}^2}{N}$$

(14)

$$Los{s}_{ssim}=\frac{(2{\mu }_{t_p}{\mu }_G+c_1)\left(2{\sigma }_{t_{p}G}+c_2\right)}{{\left(({\mu }_{t_p}\right)}^2+{\left({\mu }_G\right)}^2+c_1)\left({\left({\sigma }_{t_p}\right)}^2+{\left({\sigma }_G\right)}^2+c_2\right)}$$

(15)

$${\sigma }_{t_{p}G}=\frac{1}{N-1}{{\displaystyle \sum }}_{i=1}^N\left(t_p\left(x_i\right)-{\mu }_{t_p}\right)\left(G\left(x_i\right)-{\mu }_G\right)$$

(16)

where $Los{s}_p$ is the proposed loss function, $Los{s}_{mse}$ is the MSE loss function, and $Los{s}_{ssim}$ is the SSIM loss function. By multiplying $Los{s}_{ssim}$ with ${Los}_{mse}$, training can be reflected by characteristic of a ${Los}_{ssim}$ and ${Los}_{mse}.{\mu }_{t_p}$ is the average $t_p\left(\mathrm{proposed} \mathrm{transmission} \mathrm{map}\right)$, ${\mu }_G$ is the average ground truth $G$, ${\sigma }_{t_{p}G}$ is the covariance of $t_p\left(x_i\right)$ and $G\left(x_i\right)$ [33], $N$ is the number of elements, $e(.)$ is the error, $c_1$, $c_2$ are variables to stabilize the division [33], and $\phi$ acts as a controlling factor.

2.4. Comparison of Transmission Maps

This section compares the generated transmission map with existing transmission maps (He et al. [5], Santra et al. [19], Ren et al. [17], and Lee [7]) using the D hazy dataset [28]. As expressed in Equation (9), the transmission map can be estimated using the scattering parameter. The transmission maps were obtained using a range of scattering parameter sets [1, 0.95, …, 0.75, 0.7]. Figure 5 shows a comparison of existing methods and the proposed method. To make the ground-truth image, the depth map was used with the scattering parameter sets to 0.75. Compared with the ground-truth image (scattering parameter (β = 0.75), the transmission map generated using He et al.’s [5] and Santra et al.’s [19] method was poorly estimated. In contrast, the transmission map estimated using the proposed method was well generated.

2.5. Training Environment

In order to learn the transmission map adaptively, this study used diverse scattering parameters. The training parameters were as follows: learning rate = 0.0001, batch size = 8, validation size = 8, number of workers = 4, grad clip norm = 0.1, number of epochs = 20, and optimizer = Adam [34]. The hardware specification was as follows: Geforce RTX 2060 12 GB, Geforce GTX 1660 super 6 GB, Intel^® core™ i7-8700 CPU @ 3.20 GHz, and 32 GB RAM. The training dataset used was D hazy [28], consisting of 1449 synthetic hazy images, ground-truth images, and depth maps. To train the transmission map, 1305 images were used, while 144 images were used for validation. Moreover, in order to perform a test of the trained transmission map, WEAPD [35] is used: this has real-world sandstorm images, including 692 images. To estimate the accuracy of training, the structural similarity index measure (SSIM) was applied [33]. The images in Figure 6 show the score of the proposed loss function and accuracy score, according to epoch. Successful training yields an SSIM score [33] close to one. As shown in Figure 6, the score of the proposed loss function and accuracy gradually converged during training because the bounce was too small.

2.6. The Recombined Image

The degraded sandstorm image was corrected using the proposed balancing method. As the balanced image had similar features to a hazy image, a dehazing procedure was applied using multiscale CNN. The recombined image can be expressed as [1,2,3,4,5]:

$$J^c\left(x\right)=\frac{I_b^c\left(x\right)-A_b^c}{\max \left(t_p\left(x\right),t_0\right)}+A_b^c$$

(17)

where $J^c\left(x\right)$ is the recombined image, $A_b^c$ is the backscattered light according to He et al.’s method [5], $I_b^c\left(x\right)$ is the color-balanced image, $t_p\left(x\right)$ is the proposed transmission map, and $t_0$ = 0.1. To refine the enhanced image, guided image filtering was applied, as expressed below [13,25].

$$I_g^c\left(x\right)=G_f\left(J^c\left(x\right), k, eps\right)$$

(18)

$$J_{ref}^c\left(x\right)=\left(J^c\left(x\right)-I_g^c\left(x\right)\right)·\gamma +I_g^c\left(x\right)$$

(19)

where $I_g^c\left(x\right)$ is the guided filtered image, $G_f^{}(.)$ is guided filter, $k$ = 16, $eps$ = ${0.1}^2$, $\gamma$ = 9, and $J_{ref}^c\left(x\right)$ is the refined image.

Figure 7 shows a comparison of He et al.’s method [5], Lee’s method [7], and the proposed method. As shown in Figure 7, in contrast to the proposed method, the image enhanced using He et al.’s [5] method had a ringing and block effect in some areas, highlighting its good performance in sandstorm image enhancement.

3. Summary

This section shows the summary of the proposed algorithm to enhance the color-casted sandstorm image naturally.

Table 1. Summary of the proposed algorithm.

Input: Sandstorm image, $I\left(x\right)$

Calculate the image-adaptive eigenvalue using (6), obtain the ${\mathsf{\lambda }}_{\alpha }^c$

Calculate the image-adaptive color-balancing parameter using (7), obtain the ${\beta }^c$

Make the color-balanced image using (5), obtain the $I_b^c\left(x\right)$

Train the transmission map based on (11), obtain the $t_p\left(x\right)$

Calculate the loss function using (12) and (13), obtain the ${Los}_p$

Recombine the enhanced sandstorm image using (17), obtain the $J^c\left(x\right)$

Refine the improved sandstorm image using (18) and (19), obtain the $J_{ref}^c\left(x\right)$

Output: Enhanced image, $J_{ref}\left(x\right)$

shows the procedure of the proposed method. As shown through Table 1, this paper to enhance the color-casted sandstorm image naturally, first obtain the color-balanced image using Equations (5)–(7). The color-balanced image has hazy or dusty feature. Therefore, to enhance the hazy image a dehazing procedure is applied using CNN with Equations (11)–(13). The image is recombined with color-balanced image and generated transmission map using Equation (17). Moreover, to refine the image, a refining procedure is applied using Equations (18) and (19).

4. Experimental Result

This paper proposes a degraded sandstorm image enhancement method using image-adaptive eigenvalues and brightness-adaptive dark channel networks. As shown in Figure 2 and Figure 7, the proposed method performed well in sandstorm image enhancement.

This section compares existing state-of-the-art methods and the proposed method according to subjective and objective measures. The subjective measure compared color balancing and the enhanced image using a weather phenomenon database (WEAPD) [35].

4.1. Color Balance Comparison

This section compares color-balancing performance of the state-of-the-art methods and the proposed method. Figure 8 and Figure 9 show multiple degraded sandstorm images and enhanced images in various circumstances. Shi et al. [9,12] improved degraded sandstorm images using the mean shift of color components. These methods perform well in color balancing but introduce artificial color components in the case of greatly degraded sandstorm images. Al Ameen [11] balanced color channels using a constant gamma value, which introduces artificial colors, as this method is not able to reflect the image’s features adaptively. Lee [7] improved the degraded sandstorm image’s color components. However, in some images, an overflow region appeared due to the large difference in abundance of red and blue channels. On the other hand, the images enhanced using the proposed method had no color shift or artificial color components. As shown in Figure 8 and Figure 9, the proposed method could adequately enhance distorted sandstorm images in various circumstances.

4.2. Comparison of Enhanced Images

This section compares the images enhanced using various state-of-the-art methods and the proposed method. Because sandstorm images and hazy or dusty images are obtained similarly, this paper included state-of-the-art dehazing methods for comparison. Figure 10 and Figure 11 show variously degraded sandstorm images in diverse circumstances enhanced using state-of-the-art methods and the proposed method. He et al.’s [5] method is frequently used in dehazing; however, in some cases, bright regions such as the sky exhibited artificial color components due to the improperly estimated transmission map. Meng et al.’s method [6] has no color-balancing procedure; thus, enhanced images had a color shift. Al Ameen’s [11] method also had an artificial color shift due to the lack of an image-adaptive balancing procedure. Lee’s [7] method resulted in overflow regions because of the large difference in red and blue channel distribution. Shi et al.’s [9,12] methods resulted in an artificial color shift in the case of greatly distorted sandstorm images. Ren et al.’s [17] method has no color-balancing procedure, resulting in the enhanced image having shifted color components. Santra et al.’s [19] method has no color-balancing procedure, leading to artificial shifted color components in the enhanced image. Li et al.’s [18] method has no color correction procedure, leading to shifted color components in the enhanced image. Gao et al. [8] method has less color shifted enhanced image meanwhile the enhanced image using this method still dimmed. On the other hand, images enhanced using the proposed method had no color shift or artificial effect.

As shown in Figure 10 and Figure 11, images enhanced using the proposed method had no artificial effect, highlighting its good performance in sandstorm image enhancement.

4.3. Objective Comparison

Distorted sandstorm images were balanced using the proposed method, including a dehazing procedure based on multiscale CNN. Figure 5 shows the transmission map generated using the proposed method and state-of-the-art methods. As shown in Figure 5, the proposed method could properly generate an image-adaptive transmission map. Therefore, to compare the transmission map generated using the proposed method and other methods, the structural similarity index measure (SSIM) [33] and mean-squared error (MSE) measure were used. The reference transmission maps were generated using a depth map with various scattering parameters. The range of scattering parameters was [1, 0.7] with a 0.05 interval. If the generated data are similar to the reference data, the SSIM score [33] is high and the MSE score is low.

Table 2. Comparison of transmission maps according to MSE scores (a lower score indicates the more similar).

MSE (AVG)	Beta
	1	0.95	0.9	0.85	0.8	0.75	0.7	AVG	STD
He et al. [5]	0.071	0.077	0.083	0.091	0.098	0.107	0.116	0.091	0.015
Lee [7]	0.019	0.021	0.023	0.026	0.029	0.032	0.037	0.027	0.006
Ren et al. [17]	0.049	0.044	0.039	0.033	0.029	0.024	0.020	0.034	0.010
Santra et al. [19]	0.105	0.112	0.120	0.127	0.136	0.145	0.155	0.129	0.017
PM	0.010	0.010	0.011	0.012	0.013	0.015	0.018	0.013	0.003

and

Table 3. Comparison of transmission map according to SSIM [33] scores (a score closer to one indicates the more similar).

SSIM (AVG)	Beta
	1	0.95	0.9	0.85	0.8	0.75	0.7	AVG	STD
He et al. [5]	0.765	0.758	0.751	0.744	0.738	0.731	0.724	0.744	0.013
Lee [7]	0.921	0.920	0.918	0.916	0.914	0.911	0.908	0.915	0.004
Ren et al. [17]	0.906	0.913	0.921	0.928	0.934	0.941	0.947	0.927	0.014
Santra et al. [19]	0.751	0.743	0.736	0.728	0.720	0.712	0.704	0.728	0.016
PM	0.916	0.917	0.918	0.918	0.917	0.916	0.915	0.917	0.001

show the average SSIM [33] and average MSE scores for the D hazy dataset [28]. As shown in Table 2, the transmission map generated using He et al.’s [5] method had a lower MSE than that generated using Santra et al.’s [19] method. The transmission map generated using Lee’s [7] method had a lower MSE score compared to other methods. The transmission map generated using Ren et al.’s [17] method had a higher MSE score than that generated using Lee’s [7] method. On the other hand, the transmission map generated using the proposed method had the lowest MSE score. Table 3 shows the average SSIM scores [33]. The transmission map generated using Ren et al.’s [17] method had a higher SSIM score than that generated using He et al.’s [5] and Santra et al.’s [19] methods. The transmission map generated using Lee’s [7] method had a higher SSIM score than that generated using He et al.’s [5] and Santra et al.’s [19] methods. The transmission map generated using He et al.’s [5] method had a lower SSIM score than that generated using Lee’s [7] and Ren et al.’s [17] methods. On the other hand, the transmission map generated using the proposed method had the higher SSIM score than that of He et al. [5], Lee [7], and Santra et al. [19] methods.

As shown by the comparison of MSE and SSIM [33] scores, the transmission map generated using the proposed method performed well in various circumstances.

The differences between the generated images and ground-truth images are shown in

Table 4. Comparison of transmission maps according to AD [36] scores (a lower score indicates the more similar).

AD (AVG)	Beta
	1	0.95	0.9	0.85	0.8	0.75	0.7	AVG
He et al. [5]	0.226	0.238	0.250	0.263	0.276	0.289	0.303	0.264
Lee [7]	0.083	0.095	0.107	0.120	0.133	0.147	0.161	0.121
Santra et al. [19]	0.301	0.313	0.325	0.338	0.351	0.364	0.378	0.339
Ren et al. [17]	−0.168	−0.156	−0.143	−0.131	−0.118	−0.104	−0.090	−0.130
PM	0.026	0.038	0.051	0.063	0.076	0.090	0.104	0.064

,

Table 5. Comparison of transmission maps according to SC [36] scores (a lower score indicates the more similar).

SC (AVG)	Beta
	1	0.95	0.9	0.85	0.8	0.75	0.7	AVG
He et al. [5]	1.948	2.012	2.080	2.152	2.227	2.307	2.390	2.159
Lee [7]	1.284	1.324	1.366	1.410	1.457	1.505	1.556	1.415
Santra et al. [19]	3.073	3.171	3.275	3.383	3.497	3.617	3.743	3.394
Ren et al. [17]	0.681	0.701	0.723	0.746	0.769	0.794	0.820	0.748
PM	1.104	1.139	1.176	1.214	1.254	1.297	1.342	1.218

,

Table 6. Comparison of transmission maps according to NK [36] scores (a higher score indicates the more similar).

NK (AVG)	Beta
	1	0.95	0.9	0.85	0.8	0.75	0.7	AVG
He et al. [5]	0.695	0.683	0.671	0.658	0.646	0.634	0.622	0.658
Lee [7]	0.879	0.865	0.851	0.837	0.823	0.809	0.795	0.837
Santra et al. [19]	0.563	0.554	0.545	0.536	0.526	0.517	0.508	0.536
Ren et al. [17]	1.203	1.186	1.169	1.152	1.135	1.118	1.101	1.152
PM	0.950	0.936	0.921	0.907	0.892	0.878	0.863	0.907

,

Table 7. Comparison of transmission maps according to NAE [36] scores (a lower score indicates the more similar).

NAE (AVG)	Beta
	1	0.95	0.9	0.85	0.8	0.75	0.7	AVG
He et al. [5]	0.342	0.353	0.363	0.374	0.385	0.396	0.406	0.374
Lee [7]	0.143	0.150	0.160	0.172	0.186	0.200	0.214	0.175
Santra et al. [19]	0.442	0.452	0.461	0.471	0.481	0.490	0.500	0.471
Ren et al. [17]	0.286	0.264	0.242	0.221	0.200	0.180	0.160	0.222
PM	0.104	0.100	0.101	0.106	0.114	0.127	0.140	0.113

and

Table 8. Comparison of transmission map according to MD [36] scores (the lower score is the more similar).

MD (AVG)	Beta
	1	0.95	0.9	0.85	0.8	0.75	0.7	AVG
He et al. [5]	0.597	0.606	0.615	0.624	0.634	0.644	0.654	0.625
Lee [7]	0.386	0.392	0.397	0.402	0.408	0.414	0.420	0.403
Santra et al. [19]	0.642	0.647	0.652	0.658	0.663	0.670	0.676	0.658
Ren et al. [17]	0.130	0.131	0.132	0.133	0.134	0.136	0.137	0.133
PM	0.314	0.319	0.324	0.329	0.335	0.340	0.346	0.330

as a function of the normalized cross-correlation (NK), average difference (AD), structural content (SC), maximum difference (MD), and normalized absolute error (NAE) [36]. Low scores for these AD, SC, MD, and NAE measures indicate similarity to the ground-truth image. Moreover higher score for NK measure indicates similarity to the ground-truth image. He et al.’s [5] and Santra et al.’s [19] methods had higher scores than the other methods, indicating a low similarity to the ground-truth image. In contrast, Ren et al.’s [17] method scored lower than other methods in terms of AD and NAE. On the other hand, the proposed method scored lowest in terms of NAE.

The enhanced sandstorm images were compared subjectively in Figure 10 and Figure 11. The learned transmission map was compared in Figure 5 and Table 2, Table 3, Table 4, Table 5, Table 6, Table 7 and Table 8. As can be seen, the proposed method enhanced the distorted sandstorm image naturally. This section analyzes the results objectively using two metrics: underwater image quality measure (UIQM) [37] and natural image quality evaluator (NIQE) [38]. The UIQM [37] is used in underwater image assessment. Because sandstorm images and underwater images have similar features such as the attenuation of the color channel and the color veil, this measure is frequently adopted in sandstorm image assessment. A high UIQM score [37] indicates a well-enhanced in terms of the colorfulness, sharpness, and contrast. A low NIQE [38] measure indicates a well-enhanced and natural-looking image.

Table 9. Comparison of UIQM scores [37] for Figure 10 (a higher score indicates a better enhancement).

	[5]	[18]	[11]	[8]	[6]	[9]	[12]	[17]	[7]	[19]	PM
	0.765	0.721	0.975	0.511	0.827	0.997	0.729	0.713	1.368	0.690	1.611
	0.868	0.908	0.941	0.657	1.109	0.816	0.917	0.983	1.502	0.862	1.716
	1.014	0.980	0.908	0.746	1.002	0.984	0.909	0.888	1.638	0.888	1.686
	0.842	0.938	0.995	0.735	0.969	0.786	0.943	0.918	1.650	0.875	1.685
	0.568	0.458	0.855	0.417	0.626	0.782	0.538	0.536	0.871	0.472	1.166
	0.838	0.923	0.985	0.723	0.935	0.698	1.035	0.946	1.865	0.943	1.731
	1.020	1.031	1.151	0.668	1.349	1.250	1.004	0.987	1.611	1.011	1.646
	0.769	0.885	0.996	0.651	0.841	0.882	0.841	0.849	1.418	0.822	1.539
	0.689	0.577	0.640	0.423	0.780	0.764	0.618	0.561	1.429	0.660	1.524
	0.750	0.747	0.994	0.561	0.946	1.118	0.810	0.709	1.435	0.787	1.714
AVG	0.812	0.817	0.944	0.609	0.938	0.908	0.834	0.809	1.479	0.801	1.602

,

Table 10. Comparison of UIQM scores [37] for Figure 11 (a higher score indicates a better enhancement).

	[5]	[18]	[11]	[8]	[6]	[9]	[12]	[17]	[7]	[19]	PM
	1.165	0.932	1.008	0.745	0.999	0.987	0.948	0.873	1.653	0.925	1.586
	0.961	1.079	1.050	0.765	1.345	0.780	1.025	1.015	1.614	1.088	1.513
	0.513	0.735	0.805	0.505	0.760	0.736	0.687	0.794	1.253	0.661	1.473
	1.085	0.905	1.057	0.741	0.911	1.059	0.963	0.874	1.600	0.887	1.588
	0.590	0.516	0.967	0.746	0.577	0.789	0.769	0.762	1.447	0.602	1.474
	0.615	0.578	0.861	0.755	0.689	0.581	0.748	0.851	1.327	0.616	1.379
	0.734	0.854	0.777	0.694	1.023	0.593	0.795	0.945	1.489	0.755	1.392
	1.081	1.070	1.251	1.028	1.081	0.909	1.159	1.174	1.651	1.055	1.575
	0.718	0.823	0.893	0.644	1.063	0.754	0.822	1.367	1.429	0.785	1.449
	0.622	0.450	0.930	0.537	0.511	0.936	0.634	0.601	1.497	0.437	1.537
AVG	0.808	0.794	0.960	0.716	0.896	0.812	0.855	0.926	1.496	0.781	1.497

,

Table 11. Comparison of NIQE scores [38] for Figure 10 (a lower score indicates a better enhancement).

	[5]	[18]	[11]	[8]	[6]	[9]	[12]	[17]	[7]	[19]	PM
	19.471	19.939	19.199	19.622	19.453	19.139	19.227	19.509	17.991	19.569	17.031
	21.172	20.302	20.406	21.284	20.344	20.800	20.298	21.204	18.552	21.178	16.098
	20.039	20.699	19.924	20.085	20.066	19.724	19.588	20.121	18.018	20.363	17.212
	19.105	19.427	19.281	19.123	19.080	19.102	18.242	19.131	16.289	19.094	16.417
	20.280	20.477	19.741	20.350	20.082	20.108	19.941	20.337	20.664	20.168	19.408
	20.286	20.521	19.985	20.094	20.093	20.060	19.470	20.215	15.264	20.115	16.084
	19.493	19.515	18.664	19.617	19.160	19.304	18.929	19.434	17.069	19.219	15.723
	19.610	19.732	19.579	19.652	19.370	19.538	19.512	19.579	18.900	19.674	18.577
	19.417	19.670	19.632	19.557	19.265	19.389	19.561	19.538	19.005	19.532	18.440
	19.600	19.657	19.353	19.749	19.511	19.367	19.381	19.646	17.805	19.634	15.206
AVG	19.847	19.994	19.576	19.913	19.642	19.653	19.415	19.871	17.956	19.855	17.020

and

Table 12. Comparison of NIQE scores [38] for Figure 11 (a lower score indicates a better enhancement).

	[5]	[18]	[11]	[8]	[6]	[9]	[12]	[17]	[7]	[19]	PM
	19.934	21.203	20.323	20.286	20.129	19.696	19.737	20.042	17.866	20.006	17.180
	18.449	18.351	17.984	18.759	17.956	18.767	17.930	18.486	15.620	18.366	15.599
	19.674	19.663	19.607	19.669	19.555	19.609	19.823	19.596	19.317	19.710	18.496
	19.796	20.722	19.682	19.859	19.937	19.296	19.456	19.916	17.848	19.954	16.849
	20.168	20.862	19.905	19.638	20.341	19.838	19.883	20.127	17.614	20.143	17.543
	19.455	19.699	19.438	19.425	19.501	19.329	19.163	19.461	17.411	19.634	16.879
	20.983	20.533	20.339	20.926	19.697	20.834	19.418	20.694	17.617	20.828	16.630
	19.185	19.369	18.751	19.081	19.152	19.242	18.270	19.198	15.331	19.199	15.722
	20.398	20.437	20.228	20.505	20.191	20.317	20.063	19.980	17.736	20.525	16.524
	19.542	19.725	19.641	19.664	19.678	19.325	19.642	19.567	18.259	19.696	18.575
AVG	19.758	20.056	19.590	19.781	19.614	19.625	19.339	19.707	17.462	19.806	17.000

show the UIQM [37] scores and NIQE [38] scores for Figure 10 and Figure 11.

Table 9 and Table 10 show the UIQM scores [37] for Figure 10 and Figure 11. He et al.’s [5] method scored higher than Gao et al.’s [8] method despite its color shift in some images. Gao et al.’s [8] method scored lower than Meng et al.’s [6] method despite its color-balancing procedure. Al Ameen’s method [11] scored higher than Gao et al.’s [8] method despite its color shift. Shi et al.’s [9] method scored higher than He et al.’s [5] and Gao et al.’s [8] methods according to the image sharpness, colorfulness, and contrast. Shi et al.’s [12] method scored lower than Shi et al.’s [9] method in some images. Ren et al.’s [17] method scored higher than Gao et al.’s method [8] despite its lack of a color correction procedure in some images. Lee’s [7] method scored higher than Gao et al.’s [8] method in some images. Santra et al.’s method [19] scored lower than Al Ameen’s [11] method despite the color shift in the latter. Therefore, the UIQM score is not an absolute measure to assess a degraded image. On the other hand, the image enhanced using the proposed method had the higher UIQM score than that of other methods.

Table 11 and Table 12 show the NIQE scores [38] for Figure 10 and Figure 11. He et al.’s [5] method scored lower than Gao et al.’s [8] method, despite the color shift. Al Ameen’s [11] method scored lower than He et al.’s [5] due to its color-balancing procedure. Li et al.’s [18] method scored higher than Gao et al.’s [8] method due to the shifted color components. Despite the presence of artificial effects and color shifts, UIQM and NIQE scores do not always line up with subjective measures; hence, these measures are not absolute in assessing the enhanced image’s quality. Meng et al.’s [6] method scored lower than Gao et al.’s method [8] despite the shifted color components. Shi et al.’s method [9] scored higher than Meng et al.’s [6] despite its lower color shift. Shi et al.’s [12] method scored lower than Al Ameen’s [11] method due to its better image-adaptive color-balancing procedure. Ren et al.’s method [17] scored higher than Shi et al.’s [12] method due to its lack of a color correction procedure. Lee’s [7] method scored lower than Al Ameen’s [11] method due to its image-adaptive color-balancing procedure. Santra et al.’s [19] method scored higher than Shi et al.’s [12] method due to its lack of an image-adaptive color-balancing procedure. On the other hand, the image enhanced using the proposed method had the lower NIQE score than that of other methods due to its lack of shifted color components and artificial effects.

Table 9, Table 10, Table 11 and Table 12 confirm that image-adaptive color balancing and dehazing are needed to naturally enhance distorted sandstorm images.

Table 13. Comparison of Figure 10 and Figure 11 and WEAPD dataset [35] according to UIQM [37] scores (a higher score indicates a better enhancement).

	[5]	[18]	[11]	[8]	[6]	[9]	[12]	[17]	[7]	[19]	PM
AVG (20)	0.810	0.806	0.952	0.663	0.917	0.860	0.845	0.867	1.487	0.791	1.549
AVG (692)	0.954	0.924	1.018	0.788	0.991	0.910	0.950	0.955	1.618	0.956	1.648

and

Table 14. Comparison of Figure 10 and Figure 11 and WEAPD dataset [35] according to NIQE [38] scores (a lower score indicates a better enhancement).

	[5]	[18]	[11]	[8]	[6]	[9]	[12]	[17]	[7]	[19]	PM
AVG(20)	19.803	20.025	19.583	19.847	19.628	19.639	19.377	19.789	17.709	19.830	17.010
AVG(692)	19.826	20.190	19.715	19.859	19.732	19.707	19.328	19.876	17.504	19.836	17.129

show the average UIQM [37] and NIQE [38] scores using the WEAPD dataset [35]. Existing dehazing methods have no color-balancing procedure. Therefore, enhanced sandstorm images have shifted color components leading to a low UIQM score and high NIQE score, as they do not sufficiently and properly reflecting the image. Therefore, to enhance sandstorm images distorted by attenuation and scattering, image-adaptive color correction and dehazing are needed. Accordingly, the proposed method achieved the lower NIQE score and higher UIQM score than that of other methods, highlighting its good performance in distorted sandstorm image enhancement.

Moreover, if the proposed method is applied to sandstorm images under a computer graphics (CG) environment [39], enhanced can also be achieved because CG images consist of pixel values. Furthermore, because the training images of the proposed algorithm are obtained via data augmentation as with [40], it is able to reflect the fluctuation in real-world circumstances. Therefore, the proposed method can be applied to both real-world and CG images.

5. Conclusions

Sandstorm images have color distortion due to scattering and attenuation by sand particles with color. To enhance the color-casted sandstorm image, a color-balancing procedure is needed. Therefore, this paper proposed an image-adaptive color-balancing procedure using image-adaptive eigenvalues to reflect the image conditions. Because the balanced image has hazy characteristics, a dehazing procedure based on multiscale CNN was then applied to the balanced image. Moreover, because image datasets have insufficient transmission maps, this paper generated various transmission maps based on a depth map with various scattering parameters. The learned transmission map was able to properly reflect the various image conditions without artificial effects. The contribution of this paper is that, by applying data augmentation to obtain transmission maps, the proposed algorithm based on machine learning can adaptively generate transmission map. Moreover, by applying eigenvalue of color channel, which can universally be used in a color-balancing method. Furthermore, in the case of sparse color components, the enhanced images in this paper presented a few artificial color shifts, representing the weak point of this paper. Sandstorm images suffer from reddish and yellowish color degradation; therefore, to naturally enhance the sandstorm image, an image-adaptive color-balancing procedure is needed, which will be addressed in the future, in addition to an image-adaptive transmission network.