An Automatic Shadow Compensation Method via a New Model Combined Wallis Filter with LCC Model in High Resolution Remote Sensing Images

Abstract

Current automatic shadow compensation methods often suffer because their contrast improvement processes are not self-adaptive and, consequently, the results they produce do not adequately represent the real objects. The study presented in this paper designed a new automatic shadow compensation framework based on improvements to the Wallis principle, which included an intensity coefficient and a stretching coefficient to enhance contrast and brightness more efficiently. An automatic parameter calculation strategy also is a part of this framework, which is based on searching for and matching similar feature points around shadow boundaries. Finally, a final compensation combination strategy combines the regional compensation with the local window compensation of the pixels in each shadow to improve the shaded information in a balanced way. All these strategies in our method work together to provide a better measurement for customizing suitable compensation depending on the condition of each region and pixel. The intensity component I also is automatically strengthened through the customized compensation model. Color correction is executed in a way that avoids the color bias caused by over-compensated component values, thereby better reflecting shaded information. Images with clouds shadows and ground objects shadows were utilized to test our method and six other state-of-the-art methods. The comparison results indicate that our method compensated for shaded information more effectively, accurately, and evenly than the other methods for customizing suitable models for each shadow and pixel with reasonable time-cost. Its brightness, contrast, and object color in shaded areas were approximately equalized with non-shaded regions to present a shadow-free image.

1. Introduction

Shadows are a common phenomenon in nature when light is occluded by objects such as buildings, clouds, and trees. In the remote sensing image acquisition process, shadows also exist in images because of low sun elevation, off-nadir viewing angles, high-rise buildings, and uneven terrain. Shadows can be categorized as cast shadows and self-shadows. A cast shadow is the part of an object that is cast on the ground, while a self-shadow is the part that is not illuminated [1]. Cast shadows were the focus of the study presented in this paper. Although objects in cast shadow areas receive some scattered sunlight from the surrounding environment, their brightness is much darker than that of the surrounding non-shaded areas. As a result, information about other objects inside cast shadows is not adequately presented, which is detrimental to extracting and reusing the shaded information. In general, this influence is positively related to the cast shadow area. The area of an object shadow is related to the object size, height, and sunlight direction, and greater shadows usually cause greater radiometric information reduction. In an image, a cloud shadow is often much larger than the shadow of a ground object [2,3]. In city construction, high-rise buildings always post larger shadows that occlude the adjacent roads and buildings. Moreover, with the development of spatial resolution of remote sensing images, this information reduction is more serious in image interpretation and affects image applications in other mapping and surveying processes. Thus, it is of great interest in the image reconstruction research arena to find a way to compensate for shaded information in remote sensing images to recover the lost information before beginning image processing steps. This compensated information can be further used in land cover classification [4], mapping, object recognition [5], etc. to improve the precision of the results.

There are two types of shadow automatic compensation methods for remote sensing image: image enhancement, compensation models. In the early 2000 s, image enhancement principles, which include Linear Correlation Correction (LCC) [6], Retinex [7,8], and histogram matching [9], were applied to shadow removal for the purpose of improving the brightness and contrast of the shadow areas in images [10,11,12,13,14]. Tsai [15] utilized the invariant color property in the shadow area to detect shadow pixels and compensated for the shaded information loss based on histogram matching. Jian Yang et al. [16] proposed an approach for global shadow compensation that utilized fully constrained linear spectral unmixing. Nair et al. [17] presented a machine-learning algorithm and an Enhanced Streaming Random Tree (ESRT) model for image segmentation and classification to extract shadow areas, after which color chromaticity and morphological processing were performed to remove shadows. Vicente et al. [18] designed a novel shadow detection and removal method based on leave-one-out optimization. First, they extracted the shadows and identified their neighboring lit regions. Next, they conducted histogram matching of the I component between the shadow regions, and the lit area was utilized to restore the shaded information. In summary, the compensation effect of their approach is related to the capability of the image enhancement algorithm and often relies on manual expertise to determine suitable parameter values. Nevertheless, it is challenging to achieve adaptive processing with this method, depending on the degree of shadow occlusion.

Thereafter, a series of compensation models based on the Gamma correction method, color constancy principle, and other methods were proposed [19,20,21]. These methods combined information from shaded areas and non-shaded areas to establish suitable compensation models. Wan et al. [22] took advantage of the Gamma correction principle to treat shadows as a multiplicative noise source. Their noise influence coefficient was related to the grayscale of the original image, and after this process, the radiation was corrected by the exponential function. Mo et al. [23] proposed an object-oriented automatic shadow detection method and a shadow compensation method by region matching based on Bag-of-Words. The I component of the shadow pixels is primarily corrected by the matched region pairs. Then, the final compensation result is heightened by the overall mean and variance of the shadow and non-shadow regions. These models generally involve many parameters; and the accuracy of the parameter values is relevant to the quality of the compensation results. However, most of the values are determined by manual expertise; and a need, therefore, exists for a way to automatically calculate suitable parameter values in shadow compensation.

All in all, the above methods cannot compensate for the information to the degree that it is the same as the non-shadow area. In addition, they are not self-adaptive enough to change the model parameters to gain the most suitable restoration, and the object’s color often deviates from the original color. Therefore, an appropriate compensation model that can improve the brightness and contrast effects similar to the non-shadow regions would be helpful. The original Wallis filter is usually used for image dodging to recover an uneven color. It typically can complement the lost color with an image contrast extension coefficient and a brightness coefficient for targeted adjusting. Yet, it cannot be used in shadow compensation directly. Since the LCC model is useful in shadow compensation, the study presented in this paper designed a compensation model based on the Wallis filter and the LCC model. Additionally, an automatic parameter calculation strategy and a final compensation combination with local information based on the designed model are discussed in detail.

2. Related Works

Most of the primary shadow compensation principles for cloud shadows and ground object shadows are similar. The image mosaic principle is often used for cloud shadow removal. However, when cloud shadows occur in high-resolution images, the typical shadow compensation methods can restore the shaded information. The shadows in high-resolution images are the primary concern of this study.

The effects of automatic shadow compensation methods depend on the capability of the compensation principles and the accuracy of the relevant parameters. Among the image enhancement principles, LCC has been shown to be more productive with fewer parameters, utilizing the target mean and variance as well as an intensity coefficient. LCC is widely used in automatic shadow compensation studies [24,25]. Chen et al. [26] used LCC to compensate for the shadow area by combining the mean and variance in the shadow and non-shadow areas with a certain compensation coefficient. The target mean and variance were obtained from the non-shadow areas; however, they applied a certain coefficient value for all the shadow regions, and their method cannot adjust based on the shadow difference. Mostafa et al. [27] detected shadows and segmented images into regions; and the shadow restoration was carried out for each region based on the degree of correspondence between the shadow and neighboring non-shadow regions and the LCC principle. Liang et al. [28] applied the LCC method to compensate for those cloud shadows which cannot be removed by image mosaic; and referring to compensation model methods, the model parameters have a vital effect on the compensation results. Zhang et al. [29] designed a cubic polynomial nonlinear compensation model and adopted the Inner-Outer Outline Profile Line (IOOPL) matching method to adaptively compensate for the shaded information. The IOOPLs were obtained for the boundary lines of shadows; and shadow removal was then performed according to the homogeneous sections through IOOPL similarity matching to provide model parameters. Friman et al. [30] implemented adaptive compensation for shadows and utilized the least-squares method to determine the parameters of the brightness correction model based on the simulated shadow cast model.

In summary, the critical principle for automatic shadow compensation is the automatic calculation of the compensation parameter value. An efficient compensation model and a reasonable parameter calculation strategy are both essential in performing accurate shadow removal. To fill this need, our method strengthened our model’s capability and designed an efficient parameter calculation strategy to improve its shadow compensation. The main highlights of our work can be summed up as follows:

(1)

By taking full advantage of Wallis filter with adjustable coefficients for contrast and brightness enhancement as well as the useful LCC model in shadow compensation, we propose a compensation model by introducing two useful intensity and stretching coefficients based on Wallis filtering and LCC model. The capability of enhancing the contrast and brightness is strengthened significantly. Then it can be applied to shadow compensation more effectively.

(2)

Customize the shadow compensation model for the pixels in each shadow region by automatic parameter calculation and a compensation combination strategy. First, the compensation parameters are calculated using automatic feature points selection and matching for each shadow area. Then, the local window information of every pixel is considered by the combination strategy. Then, the adaptive compensation model is implemented so that they are suitable to recover the shaded information more flexibly and evenly.

3. Materials and Methods

Our model compensates for the shaded information automatically in monocular true color images in Red, Green, and Blue (RGB) color space. The flow chart of our automatic shadow compensation process proceeds as follows and is also is illustrated in Figure 1. Initially, the image in RGB space is transformed into a normalized Hue, Saturation, and Intensity (HSI) color space to gather the Hue (H), Saturation (S), and Intensity (I) components. $I$ is the only component compensated for. Then, typical shadow spectral features, such as low brightness and high normalized blue component, are employed to detect the shadows; and each shadow and non-shadow area is optimized and confirmed by their morphology. The means of the Red (R), Green (G), Blue (B) components in shadow areas and non-shadow areas are also calculated, respectively, to acquire their difference, including $\Delta R$, $\Delta G$, and $\Delta B$, which are used to correct the results later in the process. The mean and standard deviation of component I of each shadow and non-shadow area are calculated. Then, the feature points around the shadow boundaries are extracted and matched to calculate the values of the unknown compensation parameters. The mean and variance of component I of each shadow and non-shadow area and the compensation parameters then are the input to build a regional improved Wallis model. Meanwhile, the mean and variance of the local window centered on each shadow pixel are calculated to establish the window compensation model. Then, the original I is heightened by combining the regional compensation with the local window compensation. In the final step, the new I, the initial H, and S are converted into RGB color space; and $\Delta R$, $\Delta G$, and $\Delta B$, which represnest the difference in R, G, and B between the shadow area and non-shadow area, are subtracted from the converted R, G, B components later on so that the colors in the shadow areas are better matched with their original colors. Each step of our methodology is discussed in detail in the subsequent sections.

3.1. Shadow Detection

Before shadow compensation takes place, the shadow areas must be detected. Shadows generally have some typical spectral features such as low brightness, high hue, and high normalized blue component B′ defined in Equation (1), and low normalized green component G′ in Equation (2). These simple features cannot be combined well to detect all the shadows. Therefore, some complex signatures, such as Q defined in Equation (3), A [31] described in Equation (4) and the Morphological Shadow Index (MSI) [32,33] define in Equation (5) are applied to identify the detect condition formula for cloud shadows and ground shadows, respectively. Q and A are designed by extending the signature difference between shadow and non-shadow based on B′, I, and G′. MSI is developed based on the Differential Morphological Profiles (DMP) of black top-hat transformed data.

$$B^{\prime }=B/\left(R+G+B\right)$$

(1)

$$G^{\prime }=G/\left(R+G+B\right)$$

(2)

$$Q=B^{\prime }-I$$

(3)

$$A=\{\begin{array}{c}2B^{\prime }-I-G^{\prime }, G^{\prime }\le T_{G^{\prime }}\\ 2B^{\prime }-I-2G^{\prime }, G^{\prime }>T_{G^{\prime }}\end{array}$$

(4)

$$MSI=\frac{{\displaystyle \sum }DM{P}_{\mathrm{B}-\mathrm{TH}}\left(d,s\right)}{D·S}$$

(5)

where R, G, and B are red, green, and blue components in RGB space, respectively. B′ and G′ are the normalized blue component and the normalized green component, respectively. I is the intensity component in HSI space. $T_{G^{\prime }}$ is the Otsu threshold of G′. s and d indicate the length and direction of the linear structure element (SE). $DM{P}_{\mathrm{B}-\mathrm{TH}}\left(d,s\right)$ is the value in DMP when using the SE(d, s). D and S denote the numbers of directionality and scale of the profiles, respectively.

Generally, the ground object shadow and cloud shadow have some similarity and difference in spectral features. For instance, the ground object shadows have larger MSI values, while the cloud shadows have smaller amounts. Then, combined with the automatic Otsu threshold strategy [34], the ground object shadow and the cloud shadow are detected using Equations (6) and (7), respectively.

$$C_{\mathrm{GOSD}}=\left\{\left(i,j\right)|\left(B^{\prime }\left(i,j\right)>T_{B^{\prime }}\&\&I\left(i,j\right)<T_I\right)\Vert \left(Q\left(i,j\right)>T_Q\&\&G^{\prime }\left(i,j\right)<T_{G^{\prime }}\right)\Vert A\left(i,j\right)>T_A\Vert MSI\left(i,j\right)>T_{MSI}\right\}$$

(6)

$$C_{\mathrm{CSD}}=\left\{\left(i,j\right)|I\left(i,j\right)<T_I\Vert G\left(i,j\right)<T_G\Vert MSI\left(i,j\right)<T_{MSI}\right\}$$

(7)

where C_GOSD and C_CSD are the pixels set of ground object shadow and cloud shadow. G(i, j) represents the value of the green component in pixel (i, j). I(i, j) represents the value of intensity component I in HSI space in pixel (i, j). B′ (i, j) and G′ (i, j) represent the value of the nominalized blue and nominalized green component in pixel (i, j), respectively. Q(i, j), A(i, j), and MSI(i, j) are the corresponding value of the complex features Q, A, and MSI in pixel (i, j), respectively. $T_{B^{\prime }}$, $T_I$, $T_Q$, $T_{G^{\prime }}$, $T_A$, $T_{MSI}$, and $T_G$, indicate the Otsu thresholds of B′, I, Q, G′, A, MSI, and G components, respectively.

Lastly, some scattered and small shadow areas are removed, and some shadow holes are filled using mathematic morphological operators such as erasing, dilating, opening, and closing. Then the final shadow regions are more complete for use in further compensation.

3.2. Shadow Compensation Model Based on Wallis Filter and LCC Model

The original Wallis filter is commonly used for image dodging to recover an uneven color. It typically can complement the lost color, but it is not sufficient enough to recover the shaded information caused by shadows. Hence, this study improved the original Wallis model by introducing the intensity coefficient and stretching coefficient to promote brightness and contrast that is more effective for shadow compensation.

The general form of the Wallis filter is defined as follows:

$$g^c\left(i,j\right)=g\left(i,j\right)·r_1+r_0$$

(8)

Or another form is expressed as follows:

$$g^c\left(i,j\right)=\left[g\left(i,j\right)-{\overline{m}}_g\right]\frac{\left(c·{\sigma }_f\right)}{\left(c·{\sigma }_g+\frac{{\sigma }_f}{c}\right)}+b·{\overline{m}}_f+\left(1-b\right)·{\overline{m}}_g$$

(9)

where, $g^c\left(i,j\right)$ and $g\left(i,j\right)$ represent the target image and the original image, respectively. The parameters $r_1$ and $r_0$ are the multiplicative coefficient and the additive coefficient, respectively, where $r_0=b·{\overline{m}}_f+\left(1-b-r_1\right)·{\overline{m}}_g$ and $r_1=\left(c·{\sigma }_f\right)/\left(c·{\sigma }_g+{\sigma }_f/c\right)$. ${\overline{m}}_g$ and ${\sigma }_g$ represent the mean and standard deviation of the component in the local area around the pixel (i, j). ${\overline{m}}_f$ and ${\sigma }_f$ are the target mean and standard deviation for this area, respectively. c (c∈[0,1]) is the image contrast extension coefficient, which is proportional to the local window size. b (b∈[0,1]) is the image brightness coefficient.

The original Wallis filter, essentially a type of image enhancement principle, is typically used in image dodging to solve the disproportionate color problems [35,36], but it is not functional for shadow compensation directly. For example, in Figure 2 two different types of shadows cast by a building and cloud were compensated for the component I with the original Wallis filter. Compared to the original shadow areas, the brightness after shadow compensation is strengthened slightly. However, the overall contrast and brightness in shadow areas are still not good enough to be the same as the non-shadow areas. Consequently, shaded information is not recovered completely. Its main weakness is that $r_1$ and $r_0$ are fixed when the target means and standard deviations of the features acquired from the non-shadow areas are combined with a certain b and c. Consequently, the filter is equivalent to a linear transformation. However, because the contrast and brightness in the shadow region are so low and the influence of c is not sufficient to extend the contrast, this linear transformation is not efficient enough to enlarge the difference to recover the information. When it is used for shadow compensation directly, it cannot be valid for more serious information loss by shadows. For this reason, a valid parameter to increase the contrast should be introduced to be used in shadow compensation.

Since the Linear Correlation Correction (LCC) model in Equation (10) is a very classic method and useful in shadow compensation. By using ${\overline{m}}_{\mathrm{NSD}}$ and ${\overline{m}}_{\mathrm{SD}}$, ${\sigma }_{\mathrm{NSD}}$ and ${\sigma }_{\mathrm{SD}}$, which represent the mean values and standard deviation of the Non-Shadow area (NSD) and the Shadow area (SD), this model can more accurately enhance the shadow information to the NSD value. However, it is also not able to adjust the contrast and brightness flexibly.

$$g^c\left(i,j\right)={\overline{m}}_{\mathrm{NSD}}+\left(g\left(i,j\right)-{\overline{m}}_{\mathrm{SD}}\right)·\frac{{\mathsf{\sigma }}_{\mathrm{NSD}}}{{\mathsf{\sigma }}_{\mathrm{SD}}}$$

(10)

Thus, in this study, a new shadow compensation model based on Wallis filter and LCC model was designed by acquiring the target value from the non-shadow area adjacent to the shadow area and adding an intensity coefficient α and a stretching coefficient β, as shown in Equation (11):

$$g^c\left(i,j\right)=\alpha ·\left({\overline{m}}_{\mathrm{NSD}}+\frac{g\left(i,j\right)-{\overline{m}}_{\mathrm{SD}}}{\beta ·r_1}-r_0·r_1\right)$$

(11)

where $\alpha$ represents the compensation intensity coefficient, $\beta$ represents the stretching coefficient, ${\overline{m}}_{\mathrm{NSD}}$ and ${\overline{m}}_{\mathrm{SD}}$ represent the mean values of the Non-Shadow area (NSD) and the Shadow area (SD), $r_1$ and $r_0$ are the multiplicative coefficient and the additive coefficient in the Wallis model.

It is worth noting that this model can capture the target information from NSD areas more precisely as well as increase the average value and gradients of the shadow component more precisely. α is more useful for reinforcing the average brightness, while β is effective for enhancing the average gradient (contrast). With the help of the specific parameters, the compensation model can be more effective in heightening the information in brightness and contrast.

3.3. Automatic Parameter Calculation Method

In order to perform reasonable compensation for each shadow area, it is necessary to customize the compensation model according to their condition. Therefore, a novel method of extracting relevant regions and matching the feature points is implemented for automatically calculating the values of the compensation parameters.

First, each shadow area and its adjacent non-shadow area as shown in Figure 3a are gained by a morphological operation to calculate ${\overline{m}}_{\mathrm{SD}}$, ${\overline{m}}_{\mathrm{NSD}}$, r₀ and r₁. The Shadow area (SD) is obtained by shadow detection initially. By applying a certain morphological dilation K₁ times to each shadow area, a ring region with width K₁ around each shadow area is acquired as its Non-Shadow area (NSD). The mean and standard deviation of the SD and NSD, ${\overline{m}}_{\mathrm{SD}}$ and ${\sigma }_{\mathrm{SD}}$, ${\overline{m}}_{\mathrm{NSD}}$ and ${\sigma }_{\mathrm{NSD}}$, are calculated. Additionally, combined with the empirical values of b and c, r₀ and r₁ are determined for each region.

Then, the feature points on the shadow and non-shadow lines are extracted and matched to automatically calculate the unknown parameters $\alpha$ and $\beta$. Since there are some ground objects divided by the shadow boundary into two parts, some feature points belonging to the same object can be found on both sides of the shadow boundary. As shown in Figure 3b, the non-shadow feature lines (green) and shadow feature lines (red) can be acquired by dilating and erasing the shadow region. Pairs of similar feature points are chosen by randomly selecting a series of points along the shadow boundary. Then, their closest feature points on the two types of feature lines are selected as the shadow feature point P_SD and the non-shadow feature point P_NSD. If they are both exposed to the same amount of sunlight, they should have similar feature values. Therefore, the feature value $g^c$ of a non-shadow feature point can be the approximate target value of its shadow similar feature value. α and β can be calculated using Equations (12) and (13) Via using similar feature point pairs, corresponding equations based on them can be constructed to estimate the unknown parameters α and β using the least-squares rule.

$$\alpha =\frac{g^c}{{\overline{m}}_{\mathrm{NSD}}+\left(g-{\overline{m}}_{\mathrm{SD}}\right)·\frac{{\sigma }_{\mathrm{NSD}}}{{\mathsf{\sigma }}_{\mathrm{SD}}}}$$

(12)

$$\beta =\frac{g-{\overline{m}}_{\mathrm{SD}}}{\left(\frac{g^c}{\alpha }-{\overline{m}}_{\mathrm{NSD}}+r_0\cdot r_1\right)·r_1}$$

(13)

where $g$ and $g^c$ are the feature values of P_SD and P_NSD, respectively. ${\overline{m}}_{\mathrm{NSD}}$ and ${\overline{m}}_{\mathrm{SD}}$, ${\sigma }_{\mathrm{NSD}}$ and ${\sigma }_{\mathrm{SD}}$, represent the mean values and standard deviations of the Non-Shadow area (NSD) and the Shadow area (SD), respectively. $r_1$ and $r_0$ are the multiplicative coefficient and the additive coefficient in the Wallis model, respectively.

Finally, after all of the parameters are calculated, the regional compensation models can be customized for each shadow. The information in the different shadow areas can be strengthened by their customized compensation models to enable access to a self-adaptive adjustment.

3.4. Final Combination with the Local Window Information

The automatic parameter calculation strategy can establish a suitable compensation model using Equation (14) for each shadow region from the regional level. However, even in a single shadow region, the shaded extent differs from the different locations in the region; especially, some objects along the shadow region boundary that could get some scattered radiometric sunlight appear lighter than the objects in the center. Therefore, the statistics information of the local window centered of the pixel is used to establish a local window compensation model using Equation (15). In order to balance the interior difference in a single shadow region, the final compensation model defined in Equation (16) was developed by combining the compensation models from the regional level and local window level.

$$g_R^c\left(i,j\right)=\alpha ·\left({\overline{m}}_{\mathrm{NSD}}+\frac{g\left(i,j\right)-{\overline{m}}_{\mathrm{SD}}^R}{\beta ·r_1^R}-r_0^R·r_1^R\right)$$

(14)

$$g_W^c\left(i,j\right)=\alpha ·\left({\overline{m}}_{\mathrm{NSD}}+\frac{g\left(i,j\right)-{\overline{m}}_{\mathrm{SD}}^W}{\beta ·r_1^W}-r_0^W·r_1^W\right)$$

(15)

$$g^c\left(i,j\right)=u·g_R^c\left(i,j\right)+\left(1-u\right)·g_W^c\left(i,j\right)$$

(16)

where $\alpha$ and $\beta$ represent the compensation intensity coefficient and the stretching coefficient, respectively, which are calculated automatically for each shadow region. ${\overline{m}}_{\mathrm{SD}}^R, r_0^R, r_1^R$ and ${\overline{m}}_{\mathrm{SD}}^W, r_0^W, r_1^W$ are the mean, the additive coefficient, and the multiplicative coefficient acquired by the statistic value of a single shadow region and local window region, respectively. $g_R^c\left(i,j\right)$ and $g_W^c\left(i,j\right)$ are the compensated I value for the shadow regional level and the local window level around pixel $\left(i,j\right)$, respectively. The local window region is a $N\times N \left(N\in \left[5,20\right]\right)$ matrix centered on the shadow pixel$\left(i,j\right)$, as shown in Figure 3a. $u \left(u\in (0,1\right))$ is the weight.

Finally, after enhancing component I of the shadow pixels and transforming HSI to RGB, the final converted R, G, and B are added by the difference between the shadow and non-shadow regions $\Delta R=R_{\mathrm{NSD}}-R_{\mathrm{SD}}$, $\Delta G=G_{\mathrm{NSD}}-G_{\mathrm{SD}}$ and $\Delta B=B_{\mathrm{NSD}}-B_{\mathrm{SD}}$, respectively, for color correction that can prevent a color deviation from the original color.

4. Experimental Results

We compared the results of our method to other reference methods on several high-resolution remote sensing images with ground object shadows and cloud shadows. These results are analyzed in the following sections.

4.1. Dataset Description and Parameters Setting

Seven typical remote sensing images with different types of shadows were utilized to test the compensation method’s efficiency, including the three images with cloud shadows in Figure 4, Figure 5 and Figure 6 and the three aerial images with ground object shadows in Figure 7, Figure 8 and Figure 9. Images 1–3 in Figure 4, Figure 5 and Figure 6 were satellite images in the United States taken from Google Earth. The cloud shadows obscured large areas of information such as trees, roads, and buildings. Consequently, they could not be used in other applications. Image 4, which was taken in the downtown area of Toronto, Canada and had a resolution of 0.12 m, contained many high-rise buildings over 20 m in height with shadows that covered many ground objects and obscured their information. Image 5 was taken over the downtown area of Toyota, Japan and had a resolution of 0.08 m. Image 6 was an International Society for Photogrammetry and Remote Sensing (ISPRS) public aerial image captured over Vaihingen in Germany; and the areas of the ground object shadows in this image were darker than the cloud shadow areas as they received less sunlight while the objects in the cloud shadow areas received some scattered sunlight. Therefore, many of the objects in the cloud shadow areas were visible while the ground object shadows were more of a challenge to remove.

Based on the detected shadows shown in Figure 4, Figure 5, Figure 6, Figure 7, Figure 8 and Figure 9b, six state-of-the-arts methods were compared to our method, as shown in Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10. The Original Wallis Compensation (OWC) algorithm is compared to show the capability promotion in shadow compensation. Then, three classical compensation models, including the Linear Correlation Correction method (LCC) of Chen [26], the Gamma Correction method (GMC) of Wan [22], the Histogram Matching method (HMT) of Tsai [15] were accomplished in comparison. Furtherly, because our method is to improve the self-adaptive ability of the compensation methods according to different shade condition, two recent methodologies which had similar research goal were compared in this paper. They were respectively the Corresponding Region compensation Method (CRM) of Mostafa [27] and the Oriented Object Polynomial Removal method (OOPR) of Zhang [29]. The difference between these methods and our method was discussed in detail.

In our method, b = 0.6, c = 0.45 were used to calculate r₀ and r₁; n = 10, u = 0.6 were used to accomplish multilevel combinations; and α and β were calculated automatically. LCC used the average α of our method, and OWC was set by the same r₀ and r₁ as our method. In GMC, γ = 1.1. In HMT, the non-shadow regions were the same as that of our method. By adopting similar parameter values, the comparison was more effective in reflecting the difference between them. To further analyze the compensation results in quantitative ways,

Table 1. The evaluation comparison of compensation results in Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10.

Image Name	Quality Index	SD	NSD	OWC	GMC	HMT	LCC	CRM	OOPR	Ours
1	B	36.2754	63.1889	58.7920	62.0299	63.2713	62.7771	73.4417	77.7327	63.7444
	T	5.1839	14.4409	5.7489	9.3673	11.3595	9.2900	8.2844	13.2295	14.2704
	QB+T	—	—	0.1866	0.0455	0.0143	0.0471	0.0790	0.0126	0.0001
2	B	57.9359	85.1085	83.2481	82.4600	91.8727	86.9860	97.1427	94.9559	87.5762
	T	7.8907	16.2861	9.6110	13.0989	13.8274	11.0122	10.2769	11.7217	15.0647
	QB+T	—	—	0.0666	0.0120	0.0081	0.0374	0.0555	0.0295	0.0017
3	B	37.0008	74.3142	61.5478	73.1817	72.7659	66.7924	86.1723	76.8533	71.7967
	T	4.9003	23.7773	4.8980	11.3497	19.7813	17.5462	21.9928	10.3542	21.7824
	QB+T	—	—	0.4423	0.1252	0.0085	0.0256	0.0070	0.1549	0.0022
4	B	56.9910	150.1633	124.6353	116.4057	133.0205	124.2297	155.2552	160.0377	149.8764
	T	10.8680	20.6191	13.3118	17.8083	21.3499	13.3118	35.5466	21.3857	21.0447
	QB+T	—	—	0.0550	0.0214	0.0040	0.0553	0.0709	0.0013	0.0001
5	B	24.6591	91.9680	65.7744	71.3056	73.0492	68.0745	81.7099	89.3095	82.5732
	T	5.5742	22.1672	10.4184	16.5486	22.1398	19.0862	15.6511	17.7781	23.2122
	QB+T	—	—	0.1576	0.0371	0.0131	0.0279	0.0332	0.0123	0.0034
6	B	23.4614	107.7722	72.7422	63.7967	80.8888	74.7187	129.5133	66.2398	96.0105
	T	10.5629	21.5307	11.1936	20.7145	25.6043	21.1387	44.5303	20.5617	23.9488
	QB+T	—	—	0.1374	0.0661	0.0278	0.0329	0.1296	0.0575	0.0062
A	B	52.7057	148.7910	107.6152	109.4551	121.2661	113.6874	105.1475	127.9258	131.1461
	T	6.7169	16.9851	9.5116	13.4378	18.8517	15.5673	12.6911	11.0926	17.1362
	QB+T	—	—	0.1053	0.0368	0.0131	0.0198	0.0505	0.0497	0.0040
B	B	52.0459	149.1248	113.8646	117.4061	136.1765	127.4515	107.5620	140.2063	139.5191
	T	4.0803	13.4322	5.3878	8.4708	13.9667	12.0483	10.5662	8.7642	13.8181
	QB+T	—	—	0.2007	0.0655	0.0024	0.0091	0.0405	0.0452	0.0013
C	B	50.3020	136.3924	116.0753	105.9343	121.3617	112.6049	142.1750	135.7543	135.5174
	T	7.0388	21.5283	8.5362	11.9807	19.4671	15.5198	25.8519	13.6585	21.2531
	QB+T	—	—	0.1932	0.0970	0.0059	0.0354	0.0088	0.0500	0.0001
D	B	54.6436	142.2926	105.7497	120.4303	121.9426	114.5220	115.9803	142.4835	129.9071
	T	4.9285	15.2054	5.9959	10.2507	15.7802	14.1726	10.9094	10.5155	15.3493
	QB+T	—	—	0.2104	0.0448	0.0063	0.0129	0.0374	0.0332	0.0021
E	B	22.1513	90.6838	72.6012	71.6895	81.1097	76.1222	73.8717	91.9199	88.5145
	T	2.8182	20.4905	8.7832	14.8224	20.3036	18.6539	16.3324	11.4661	20.5716
	QB+T	—	—	0.1722	0.0394	0.0031	0.0098	0.0232	0.0798	0.0002
F	B	22.1948	116.9342	84.3402	71.4961	95.5993	87.6406	112.3088	86.5574	102.5739
	T	6.4213	18.4400	7.3065	14.6794	21.0539	17.7352	30.7426	16.6335	17.2710
	QB+T	—	—	0.2132	0.0710	0.0145	0.0209	0.0630	0.0249	0.0054
Average QB+T		—	—	0.1915	0.0581	0.0099	0.0268	0.0557	0.0500	0.0021

shows the original shadow brightness, the target value in the non-shadow area, the compensated value, and the compensation quality index of Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10.

4.2. Precision Evaluation Criteria

The compensation quality, referred to as the brightness and average gradients, which have been used in some studies, can quantitatively assess the resulting difference between compensated shadows and non-shadows.

B is the mean value of component I, as defined in Equation (17) and it reflects the brightness level of the measured area. Assume S(x,y) is the local area in the image to be analyzed and $N_S$ is the pixel number in this area. Average brightness B reflects the degree of lightness and darkness of the area [1]. T, as defined in Equation (18) represents the average gradient (contrast) and reflects the amount of image detail and clearness of an image.

$$B=\frac{1}{N_S}{\displaystyle \sum }_{\left(i,j\right)\in S}I\left(i,j\right)$$

(17)

$$T=\frac{1}{N_S}{\displaystyle \sum }_{\left(i,j\right)\in S}\sqrt{\frac{1}{2}\left\{{\left[I\left(i+1,j+1\right)-I\left(i,j\right)\right]}^2+{\left[I\left(i+1,j\right)-I\left(i,j+1\right)\right]}^2\right\}}$$

(18)

where $I\left(i,j\right)$, $I\left(i+1,j\right)$, $I\left(i,j+1\right)$, and $I\left(i+1,j+1\right)$ represent the I values in the pixels $\left(i,j\right)$, $\left(i+1,j\right)$, $\left(i,j+1\right)$, and $\left(i+1,j+1\right)$, respectively. $N_S$ is the number of pixels in the shadow area.

Because the feature value in the non-shadow area can be seen as the approximate target value, (ΔB)² and (ΔT)², as defined in Equations (19) and (20) which are normalized by the non-shadow value, can represent the difference between the compensation results and the target values to evaluate the quality of the compensation results and facilitate the analysis of the effects of the model parameters. Then, in order to evaluate the total bias from the non-shadow regions, Q^B+T defined in Equation (21) can be calculated and seen as the total compensation quality. In general, lower values of Q^B+T indicate compensation results that were closer to the non-shadow area.

$${\left(\Delta B\right)}^2={\left(\frac{B-B_{\mathrm{NSD}}}{B+B_{\mathrm{NSD}}}\right)}^2$$

(19)

$${\left(\Delta T\right)}^2={\left(\frac{T-T_{\mathrm{NSD}}}{T+T_{\mathrm{NSD}}}\right)}^2$$

(20)

$$Q^{B+T}={\left(\Delta B\right)}^2+{\left(\Delta T\right)}^2$$

(21)

where B and T are the compensated brightness average and the average gradient value of the shadow area. B_NSD and T_NSD are the brightness average and the average gradient of the non-shadow area. (ΔB)² and (ΔT)² are the square of the normalized difference between the compensated value and the non-shadow area in B and T, respectively. Q^B+T is the total compensation quality.

4.3. Qualitative Comparison

The cloud shadow compensation comparison results effectively show the difference in a single shadow area. Figure 4, Figure 5 and Figure 6 show that our method obtained better results in brightness, contrast, and original colors than the other methods. OWC’s results indicate that the brightness was rising while the contrast was almost unchanging. LCC, GMC, HMT, CRM, and OOPR improved both the brightness and contrast to some extent and the original color of the objects in the shadow areas was recovered (e.g., the trees in Figure 4 are depicted in their true green color). HMT produced the best and most stable capability for improving the contrast while the other methods’ contrast indicated another problem of uneven compensation in a single shadow area. The part of the cloud shadow area that was obscured by the thicker cloud is still slightly darker than the thinner part after compensation in almost all the reference methods. In comparison, this uneven compensation problem was solved by the multilevel combination strategy. For the bare land in Figure 5, there is a small non-shadow region that is recognized as shadows. This part was easily overcompensated by other methods, but our method addressed this phenomenon because its local window information was useful in adaptively enhancing each partially shaded information to the same level as the non-shadow area. Thus, our model recovered the information as accurately as possible and showed almost no difference in comparison to the non-shadow areas.

Regarding ground object shadow compensation, our method produced better results, which indicated its ability to compensate for each shadow region. Different shadow regions were improved and were identical to their adjacent non-shadow areas, which was beneficial for avoiding over-compensation or insufficient compensation. Even though the object information shaded by the buildings was too dark to see, our method recovered the information. To depict the compensation results of our method in detail, six portions of Images 4–6 in Figure 7, Figure 8 and Figure 9 were selected, which are named A-F in Figure 10. From the figures and Table 1, it can be seen that our method produced better visual and quantitative compensation results than the other methods.

When compared to the non-adaptive methods (OWC, GMC, HMT, and LCC), our approach was able to compensate for shadows more adaptively. Although OWC significantly enhanced the brightness, the shadow areas were still unclear for less heightened contrast. GMC also improved the brightness and contrast efficiently; and while its contrast enhancement was better than OWC, it was not as good as the other methods, as shown in boxes B₂, E₂, and F₂. Because parameter γ could not be adaptive to every image and shadow area, it was hard to decide which value was best for them. Therefore, GMC’s compensation capability was not stable and self-adaptive. The effect of GMC was different among the images; and therefore, sometimes it was good, as seen in Images 2 and 5, but mostly it was not good enough. HMT produced stable compensation results and was especially good at contrast enhancement; however, sometimes it was over-enhanced and lost the original information. As can be seen in box A₃, the shadow area was over-compensated in contrast. Comparatively, LCC, as one of the most typical compensation methods, undeniably recovered the primary information in the shadow area; and while it enhanced brightness and contrast efficiently, it was unable to achieve the same levels as in the non-shadow areas. LCC also produced color deviation and uneven improvements in different shadow areas; and its results show a blue color instead of the original color. This color difference was mainly caused by its over-enhanced illumination, resulting in a higher blue component than the non-shadow area. However, the color bios in cloud shadow were less serious than the ground object shadows since the difference between the compensated shadow area and the non-shadow area was low. In comparison, the results for our method represented the original roof color better due to our color correction strategy. Additionally, the problem of uneven compensation was present in most of the compared methods (e.g., the boxes in Images C and D). Box C₄ shows that LCC experienced inefficient and uneven compensation.

Recent methods, such as OOPR and CRM, were designed to obtain self-adaptive parameters adjusted according to each shadow as well. Although they adopted their respective adaptive parameter strategies, they did not achieve stable results in the various images. OOPR utilizes a polynomial fitting compensation principle, which cannot solve the best function for each shadow region from the IPOOL strategy. As a result, OOPR improved the brightness similarity to the non-shadow area, but its contrast enhancement was inadequate due to its limited polynomial function. As shown in Image 3, the brightness was satisfied while the contrast did not meet the non-shadow level. In addition, although CRM also utilizes LCC as the compensation model, it gained the non-shadow information from the adjacent non-shadow segment areas of each shadow area. CRM produced overcompensation as shown in Image 4, Image C, and Image F. For example, the shadow in box F₅ is obviously over-compensated; and the contrast enhancement in Image F is worse than in Images C and D. When the segmentation areas were not suitable, the shadow area was over-compensated or insufficiently compensated, as shown in Image F. It is difficult to both improve the brightness and contrast at the same time in a way that is similar to non-shadow areas, which was a common problem for CRM and OOPR.

In contrast, these problems were solved by our method, since it was able to adaptively recover the shaded information as clearly and initially as it would appear in sunlight. Since our approach uses different parameters for different shadow areas, the shadow information is strengthened more reasonably according to every shadow situation. The most significant benefit of this strategy is that it prohibits over-compensation or inefficient compensation. Moreover, using the local window compensation, the details in each shadow area are more evenly promoted.

4.4. Quantitative Comparison

In the quantitative comparison, our method achieved the best compensation quality image measured by Q^B+T. The reinforcement of T was closer to the non-shadow area while the average brightness was slightly lower than that of the non-shadow area; and the total compensation quality was much better than the other methods as well. Figure 11 shows the 25 images that were tested and compares the average of ${\left(\Delta B\right)}^2$, ${\left(\Delta T\right)}^2$, and Q^B+T they achieved. The results were analyzed by combining visual and quantitative outcomes. The average Q^B+T of our method in Figure 11 and Table 1 were 0.0038 and 0.0021, respectively. Both values indicate the same conclusion, namely, that our method produced the best compensation results, followed by the HMT method with 0.0099, and then the LCC with 0.0268. The OWC method attained the worst results due to its low capability in enhancing contrast. The other methods were not very stable. Since the Q^B+T cannot reflect the color deviation and the even contrast detail in a single shadow area, the methods’ results differed visually, as shown in Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10.

In summary, the comparative results indicate that our method enlarged the shaded information from low quality to a quality that was close to the non-shaded area in brightness and contrast in a more balanced way. Additionally, our method recovered the color information and self-adaptively constructed the compensation model, resulting in more even enhancement and better recovery of the original information of the shadow objects.

4.5. Time Computation

The proposed method was implemented in a VS2019 environment using a hybrid program code based on C++ and OpenCV. All the experiments were conducted on a laptop with an Intel Core i7 CPU, 2.60 GHz, and 32 GB RAM. The computation times for all the compared methods are shown in

Table 2. Shadow compensation time comparison of different images in Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10.

Image Name	Shadow Rate (%)	Length × Width (pixel)	OWC (s)	GMC (s)	HMT (s)	LCC (s)	CRM (s)	OOPR (s)	Ours (s)
1	50.563	823×576	0.075	0.033	0.034	0.179	4.627	1.102	0.893
2	38.979	686×601	0.076	0.038	0.033	0.082	1.650	0.889	0.677
3	45.893	894×634	0.143	0.065	0.064	0.159	2.336	1.604	1.338
4	26.783	1437×937	0.297	0.172	0.186	0.375	33.469	5.034	4.688
5	22.352	937×1437	0.186	0.105	0.108	0.234	20.276	1.903	1.641
6	27.077	1004×1188	0.263	0.129	0.129	0.269	32.886	1.525	1.264

. The time of GMC and HMT was similar, and they were the shortest because they adopt the same parameter value to the whole shadow areas in an image. OWC and LCC use the mean and variance of each shadow area, which resulted in longer computation times. CRM requires segmentation results in order to obtain the non-shadow region information; therefore, it required the longest time to accomplish shadow compensation. Both our method and OOPR solved the parameter values for each shadow; but because OOPR adopts an IPOOL strategy, it is more complicated than our approach and therefore required a little more time. Although our method’s computation time was moderate but not the fastest, it accomplished a better compensation result for each shadow area.

5. Discussions

To evaluate the capability of the proposed method more specifically, we analyzed the impact of the introduced coefficient α and β on the compensation quality indexes and the effectiveness of the automatic calculation strategy.

5.1. The Positive Impact of α and β on the Proposed Model

Taking I as an example, the compensation experiment continued on the shadow image TIGS in Figure 2a and TICS in Figure 2c to verify the influence of α and β. A comparison of the compensation results by assigning different values to α and β was implemented, as visually shown in

Table 3. Comparison of results compensation for TIGS and TICS by fixing β = 10 and applying different α in Equation (5).

	α = 0.5	α = 0.75	α = 1	α = 1.25
β = 10

and

Table 4. Comparison of results compensation for TIGS and TICS by fixing α = 0.95 and applying different β in Equation (5).

	β = 5	β = 10	β = 15	β = 20
α = 0.95

. The figures in the table show how the brightness and contrast changes with the change of the parameter. When fixing β and changing α in compensation model of Equation (5), B and T are both improved with the increase of α; when fixing α and changing β, T decreases, and B remains unchanged with the rise of β.

Figure 12 describes the influence of α and β on the compensation results. Figure 12a,b show that α affects both B and T linearly and positively. However, for the values for α that were too high, the brightness and contrast compensation was excessive, while the values that were too low for α made the brightness and contrast compensation insufficient. As shown Figure 12c, with the increasing value of α, (ΔB)² and (ΔT)² decreased until they reached the lowest points; and the increasing α led to an increase in (ΔB)² and (ΔT)². Hence, selecting an optimal value for α was important to achieve a target compensation value. The best solution was to acquire the α value, where Q^B+T achieved the minimum. β had little influence on B, and it only affected the T of the shadow area nonlinearly and negatively, as shown in Figure 12d,e, respectively. Meanwhile, the impacts of β on (ΔB)² and (ΔT)² were similar to the impacts on B and T. With the increase of β, (ΔT)² decreased greatly and flattened. Therefore, β was effective for strengthening T (the contrast) more efficiently. If a suitable value of β was selected based on the lowest Q^B+T, the information about the shadow area was restored. Thus, both α and β had a positive impact on compensation quality. The lowest Q^B+T pinned down the best value of α and β, which helped to achieve brightness and contrast similar to the non-shadow target.

5.2. The Effectiveness of the Automatic Strategy for α and β Calculation

The automatic strategy for α and β calculation is important for accomplishing automatic compensation without manual direction. It also helps customize suitable compensation models for each shadow area but not for the whole image. In order to verify the effectiveness of the automatic parameter strategy, four different shadow areas were tested to compare the optimal values acquired by the minimum Q^B+T with the automatic values of α and β. The relationship between Q^B+T and α as well as Q^B+T and β, as shown in Figure 13, were used to estimate the ideal values of α and β at the minimum Q^B+T. Furthermore, the automatic values of α and β calculated by the strategy are shown in

Table 5. Comparison of α and β values calculated automatically with its ideal value of different regions.

Area Name	α Automatic Value	β Automatic Value	α Ideal Value	β Ideal Value
1	1.295	1.036	1.35	1
2	1.264	0.926	1.3	1
3	1.091	9.831	1.1	10
4	1.0863	6.321	1.05	6.5

. The comparison results show that the automatic calculation value was close to the estimated ideal value for both α and β, which indicates that the parameter calculation strategy effectively calculated the suitable values for the compensation parameters. As a result, those values were effective in customizing a suitable model for each shadow.

5.3. Validation for Color Correction

Since our method can be used to heighten any component to recover the shadow information, choosing efficient components to raise the shadow information closer to its original content is significant. In general, H, S, and I were all used together to compensate in several studies, but this approach is not effective for maintaining the original color of the shadow objects. Because it almost raises all of the components to similar values, the results show a gray color that does not reflect the original colors, as shown in Figure 14a,d. In fact, during the experiments and in our previous research [31], component I was rather efficient for cloud shadows compensation to recover the original information, leading to some color loss in ground object shadows compensation. As shown in Figure 14b,e, the results were greater with the proposed model for compensating component I; and within the building shadows, there is a color deviation from the original color that does not occur in the cloud shadow compensation results. As shown in Figure 14c,f, the results optimized by the color correction strategy show the original colors were well maintained.

6. Conclusions

This paper introduced and demonstrated the implementation of a new shadow compensation model that combines the Wallis filtering principle and LCC model by adding intensity coefficient α and stretching coefficient β to adjust brightness and contrast effectively. When combined with a parameter automatic extraction scheme based on feature point pairs, our method was found to target loss information more accurately; and the compensation parameters in our model were able to be obtained automatically. By combining the local window and a regional compensation model, using a multilevel combination strategy, the shaded information was evenly enhanced as well. As shown in this paper, when compared to other state-of-the-art methods, our contrast and brightness enhancements produced better and more consistent results for the non-shadow areas. They were able to restore the real information of the features shaded by shadows. Our model’s shadow-free, higher-quality images demonstrate its image reconstruction capabilities and its potential for use in complete land-cover or land-use map applications.