Considering the Effects of Horizontal Heterogeneities in Satellite-Based Large-Scale Statistics of Cloud Optical Properties

Abstract

This paper explores a new approach to improving satellite measurements of cloud optical thickness and droplet size by considering the radiative impacts of horizontal heterogeneity in boundary-layer cumulus clouds. In contrast to the usual bottom-up approach that retrieves cloud properties for individual pixels and subsequently compiles large-scale statistics, the proposed top-down approach first determines the effect of 3D heterogeneity on large-scale cloud statistics and then distributes the overall effects to individual pixels. The potential of this approach is explored by applying a regression-based scheme to a simulated dataset containing over 3000 scenes generated through large eddy simulations. The results show that the new approach can greatly reduce the errors in widely used bispectral retrievals that assume horizontal homogeneity. Errors in large-scale mean values and cloud variability are typically reduced by factors of two to four for 1 km resolution retrievals—and the reductions remain significant even for a 4 km resolution. The calculations also reveal that over vegetation heterogeneity-caused droplet size retrieval biases are often opposite to the biases found over oceans. Ultimately, the proposed approach shows potential for improving the accuracy of both old and new satellite datasets.

1. Introduction

Clouds constitute a critically important element of the climate system. Their complexity, however, poses formidable challenges, and clouds are widely recognized to be a key source of uncertainty in our understanding and predictions of climate [1,2,3]. As the Intergovernmental Panel on Climate Change 6th assessment report released in 2021 states, “…clouds remain the largest contribution to overall uncertainty in climate feedbacks” [4].

Due to their extensive spatial coverage, satellite observations of clouds can help to reduce current uncertainties, as they provide a rich archive for characterizing the Earth’s cloud cover [5], for evaluating the realism of clouds simulated by climate models [6], and for improving our understanding and the simulation capabilities of cloud-related processes such as aerosol–cloud interactions [7].

The critical cloud information provided by satellites includes the widely used measurements of cloud optical depth and cloud droplet effective radius. These two parameters can help, among others, to better understand cloud radiative effects [8] and to estimate both the water content of clouds and the number of cloud droplets [9,10]—which, in turn, enables further analyses, for example, of aerosol–cloud interactions [11].

Most satellite measurements of cloud optical thickness and droplet size use a bispectral technique developed in the 1970s and 1980s [12,13]. This technique combines observations of reflected sunlight at a visible or near-infrared wavelength where clouds do not absorb sunlight, and at a near-infrared wavelength where clouds absorb light at a rate that increases with droplet size. The technique, however, is based on one-dimensional (1D) radiative transfer theory and relies on the assumption of homogeneous, plane-parallel clouds. This means that, in interpreting the radiance data from a particular pixel, the method considers neither horizontal photon transport (that is, radiative interactions between nearby areas that have different cloud properties), nor the effects of unresolved small-scale variability within the pixel. Since the first studies in the 1970s [14,15], as described below, numerous theoretical and observational studies have indicated that the 1D assumption can cause substantial errors (including biases) in satellite retrievals of both optical thickness and effective radius.

Cloud optical thickness retrievals were shown to underestimate the mean optical thickness for high sun (more than ≈30° above the horizon) and overestimate it for low sun (less than ≈30° above the horizon) in theoretical studies. For high sun, thick clouds scatter light into thin areas where they can easily pass through the cloud layer [16], thus reducing the area-averaged reflectance and, hence, the retrieved optical depth. For low sun, interception by cloud sides prevents some incoming sunlight from simply passing through cloud-free gaps, thereby increasing reflectance and the retrieved optical depth [14,17,18]. Statistical analyses of satellite observations have shown that this introduces a strong solar elevation dependence into the mean optical thicknesses retrieved by the Advanced Very-High-Resolution Radiometer (AVHRR) and MODerate-resolution Imaging Spectroradiometer (MODIS) [19,20,21]. Naturally, such biases can also greatly affect the perceived latitude (or even the local time) dependence of cloud statistics (e.g., Level 3 products) obtained from polar-orbiting or geostationary satellites [22]. Finally, we note that three-dimensional (3D) radiative processes not considered in 1D retrievals distort not only the mean, but also the perceived spatial distribution of cloud optical thickness. Photon diffusion can make clouds appear too smooth [23,24,25], whereas 3D shadowing and side illumination effects can make clouds appear too rough [26,27], asymmetric [28,29,30], and anisotropic [27].

Cloud droplet effective radius retrievals were also shown to be affected by horizontal cloud heterogeneity. Theoretical studies explored these effects in detail [31,32], and satellite data analyses found systematic effects, for example, by linking cloud heterogeneity to differences between droplet sizes estimated using different MODIS wavelengths [33]. An analysis of global multiangle satellite data also indicated large biases in particle size retrievals for heterogeneous clouds [34]. Finally, we note that cloud heterogeneity distorts our view of not only mean, but also the spatial distribution of droplet radius values. Satellite-retrieved effective radii were found to be systematically larger when the cloud top was tilted away from the sun and not toward it [29,30].

Recognizing the limitations of current bispectral retrievals, researchers have explored various alternatives [35], most of which can also consider information on spatial and angular variations and even polarization. One approach seeks to iteratively adjust the estimated cloud structure in order to reduce the discrepancy between the observed cloud reflectances and the cloud reflectances simulated by a 3D radiative transfer model [36], often as part of cloud tomography [37,38,39,40,41,42]. Another approach estimates cloud properties using a neural network, machine learning, or artificial intelligence [43,44,45,46,47]. Such new approaches have yielded promising results and, indeed, revealed potential paths for great improvements in our cloud retrieval capabilities. At present, however, none of these methods have reached the mature phase of being used operationally or as a tool in studies that do not focus on retrieval methodologies, but just need accurate cloud properties. Additionally, some of these techniques require multiangle or polarization observations, which may limit their applicability to advanced satellite sensors with such capabilities.

The present study explores a new approach to improving satellite measurements of cloud optical depth and droplet size for heterogeneous clouds. The primary goal is to help improve current estimates of large-scale cloud statistics, such as the mean and standard deviation of the cloud optical thickness and effective radius over areas that are several dozen kilometers along each side. This could help to remove systematic biases from various cloud studies and could improve the accuracy of some Level 3 cloud products such as the MODIS and VIIRS (Visible Infrared Imaging Radiometer Suite) Level 3 Cloud Properties Products provided at a 1° by 1° latitude–longitude resolution [48].

Given this primary goal, the proposed approach reverses the flow of past efforts that have focused on retrievals for individual pixels, with any large-scale statistics being compiled subsequently. The proposed approach first determines the effect of 3D heterogeneity on 1D estimates of large-scale statistics of cloud parameters and then distributes the overall effects to individual pixels. Compared to the traditional “bottom-up” approach, the proposed top-down approach offers advantages such as higher computational efficiency and a better-posed inversion problem. It promises improved cloud statistics without requiring us to untangle the complex web of 3D radiative interactions that, for decades, have stymied the retrievals of heterogeneous cloud properties for individual pixels. (The terms “bottom-up” and “top-down” seek to capture the following difference: in the “bottom-up” approach, using a physics-based or empirical method, we obtain results for each pixel and then build statistics using the individual pixel values as our foundation, whereas in the “top-down” approach, we start by obtaining overall top-level statistics and then estimate the pixel values underlying these estimated statistics.)

We note that the proposed approach is inspired, in part, by past efforts to parameterize the impact of 3D radiative processes (on radiative fluxes or radiances), not for specific locations, but for heterogeneous cloud fields as a whole [49,50,51,52,53]. The closest predecessor approaches, which also considered biases in cloud retrievals, were described in [54,55].

The outline of this paper is as follows. First, Section 2 describes the dataset used in this study. Next, Section 3 describes the proposed retrieval technique, and Section 4 presents the results. Finally, Section 5 provides a brief summary and offers some concluding remarks.

2. Dataset

The proposed technique seeks to adjust (and improve) current 1D retrieval results using statistical relationships between the optimal adjustments and satellite-observable quantities. To establish and test these statistical relationships, we need three types of data, as follows: (i) heterogeneous cloud fields specified in detail, (ii) radiation fields for these clouds, and (iii) look-up tables for the 1D retrievals whose results we seek to adjust. These three types of data are described in the following subsections.

2.1. Cloud Fields

This study used the 3D liquid water content fields of liquid-phase trade cumulus clouds generated using 52 Large Eddy Simulations (LESs). The LES simulations were performed by the LES model developed by Bjorn Stevens in 2005 [56] and upgraded a few years later [57]. One half (26) of the 52 simulations were performed by Frank Evans and were described in detail in [58]. The other half of the simulations were repeat runs performed by Guoyong Wen using the same setup and inputs, but with different random perturbations to the initial atmospheric fields, which yielded completely different cloud arrangements during the LES simulation period.

Each simulation followed cloud evolution in a 20 by 20 km region through an 8 h long period. The horizontal resolution was 62.5 m, and the vertical resolution was 40 m or finer. From each 8 h simulation, we used cloud structures at 17 instances separated by half-hour intervals, during which, the arrangement of cloud elements could change drastically. At the selected instances, the cloud cover ranged from 0 to 60%, the scene-average optical thickness ranged from 0 to 15, and the cloud top altitude ranged up to 4 km. To illustrate the variety of clouds in our database, Figure 1 shows a few sample scenes.

We note that the LES-generated scenes (e.g., Figure 1) fell into the cloud organization pattern termed “sugar” [59], which occurs frequently over both land and ocean [60].

Because LES simulations are computationally expensive, we enhanced the variety of cloud structures in our database and considered four versions of each scene, with each version being rotated by 0°, 90°, 180°, or 270°. This rotation helped because, from the perspective of the incoming sunlight, the cloud arrangements (and radiative heterogeneity effects) became very different after each rotation—for example, a point in shadow may have become sunlit after a rotation. Naturally, this rotation would not make a difference for exact overhead sun, but for the solar zenith angles (≥15°) considered in this study, the rotation made a significant difference: The results shown throughout the paper rarely feature closely grouped data points that likely come from differently rotated versions of the same LES scene.

Ultimately, after all completely cloud-free instances were eliminated, the dataset contained 3176 LES scenes for use in this study.

2.2. Radiation Fields

The next step in creating our dataset was to simulate the radiation fields a satellite would observe over the LES-generated cloud fields. Specifically, the simulations provided 1 km resolution radiation fields at three wavelengths often used in cloud retrievals over land, as follows: 0.65 µm, 2.13 µm, and 11 µm. For example, these wavelengths and this resolution are fairly similar to the parameters of the MODIS and VIIRS observations used in cloud optical retrievals.

Since this study seeks only to explore a possible new approach and gauge its potential usefulness, the simulations did not need the most realistic, detailed, and all-encompassing set of scene and observation parameters. As a result, the simulations provided radiation fields only for a 0° viewing zenith angle (Θ) and three solar zenith angles (Θ₀), 15°, 45°, and 75°. Also, the simulations ignored factors such as the curvature of the Earth, gaseous absorption, and atmospheric aerosols, and used simple approximations for other environmental effects. Rayleigh scattering was considered using the formulas in [61], and a Lambertian underlying surface was assumed with albedos of 0.1, 0.3, and 0.0 at 0.65, 2.13, and 11 µm wavelengths, respectively.

As the LES simulations provided only liquid water content values and gave no information on droplet size distribution, we assumed a modified gamma drop size distribution with an effective radius (r_e) of 8 µm. We note that using a constant drop size distribution made the analysis much simpler, as we did not need to deal with ambiguities in the “correct” result that would occur if r_e varied within a pixel. (For example, vertical variations in drop size cause even perfectly accurate methods to yield different results if they weigh differently the drops occurring at various altitudes [62,63].)

The wavelength-dependent cloud droplet extinction efficiencies, single-scattering albedos, and scattering phase functions were calculated using Mie theory [64]. The 3D distribution of 0.65 µm volume extinction coefficients (β_0.65) was calculated using the formula,

$${\beta }_{0.65}\left(x,y,z,t\right)={\displaystyle \frac{3}{2}}{\displaystyle \frac{LWC\left(x,y,z,t\right)}{\rho \ast r_e}}$$

(1)

where LWC is the liquid water content provided by the LES simulation (as a function of the coordinates x, y, and z and time t) and ρ is the density of water. The extinction coefficients for other wavelengths, ${\beta }_{\lambda },$ were then calculated using the equation,

$${\beta }_{\lambda }={\beta }_{0.65}{\displaystyle \frac{Q_{\lambda }}{Q_{0.65}}}$$

(2)

where λ identifies the wavelength (2.13 µm or 11 µm) and Q is the extinction efficiency calculated using Mie theory [64]. The cloud droplet single-scattering albedos and scattering phase functions were calculated using Mie theory at all three wavelengths.

Once all the scene parameters were set, 3D radiative transfer calculations provided the radiation fields that a hypothetical satellite would observe at the three wavelengths. The radiative simulations were performed using an updated version of our Monte Carlo model that was validated through the Intercomparison of 3D Radiation Codes (I3RC) project [65,66] and used in several earlier studies [67,68]. The radiative processes were simulated using the 62.5 m resolution LES clouds, but the simulated fields of nadir radiances were degraded to a 1 km horizontal resolution, which is representative of satellite sensors such as MODIS or VIIRS. The 0.65 and 2.13 µm radiances were converted to reflectances, also called bidirectional reflectance factors (BRFs), defined as $BRF=\pi I/{\mu }_0{F}_0$, where I is the nadir radiance, µ₀ is the cosine of the solar zenith angle, and F₀ is the solar irradiance. The 11 µm radiances were converted first to equivalent brightness temperatures using Planck’s law, and then to apparent effective altitudes, assuming a 20 °C surface temperature and a 6.5 °K/km vertical temperature gradient.

The calculations simulated 400,000 photon paths for each 1 km size pixel (using the method of local estimates [69]), which ensured a sufficient computational accuracy for our purposes. The mean of the largest single-pixel uncertainty values encountered within each scene was less than 0.013 and 0.03 for the 0.65 and 2.13 µm reflectances, respectively, and about 15 m for the effective altitudes.

Finally, we note that our computational resources allowed radiation simulations for all 3176 scenes only for the 45° solar zenith angle; for the 15° and 75° solar zenith angles, we performed simulations for a randomly selected set of 1200 scenes (≈37% of the 3176 scenes).

2.3. Retrieval Tables

After simulating what a satellite would see over the LES scenes, our next task was to examine how the widely used bispectral retrieval method (based on 1D radiative transfer) would interpret the simulated observations. In practice, this meant creating look-up tables that provided cloud optical thickness (τ) and droplet effective radius (r_e) values as a function of 0.65 µm and 2.13 µm reflectances. These tables—created using the Discrete Ordinates Radiative Transfer (DISORT) model [70]—are illustrated in Figure 2.

As discussed in the next two paragraphs, the figure reveals that droplet size retrieval biases caused by subpixel variability were often opposite over vegetated areas than in the oceanic regions that were the focus of earlier studies on cloud heterogeneity effects [31,32,71,72].

Although the figure considers vegetated regions, the points marked A, B, and C illustrate the nonlinearity-caused biases that were found earlier in oceanic regions (often simulated assuming a dark, non-reflecting ocean surface). For this, let us consider a retrieval performed for a pixel that is clear in one half and is covered by a thick cloud in the other half (with corresponding reflectances identified by points A and B, respectively). A retrieval for the entire pixel will then be based on the average 0.65 µm and 2.13 µm reflectances of points A and B, which is identified as point C. The figure shows that, while the dark blue color at point B indicates that the true r_e of the cloud is 4 µm, the lighter color at point C shows that the r_e retrieved for the pixel will be closer to 8 µm. This illustrates that, in the presence of subpixel variability, the curved (nonlinear) pattern of colors identifying the retrieved r_e values will cause overestimations of the effective radius values. While 3D radiative effects such as side illumination can reverse these errors for individual pixels, overall biases always meant r_e overestimations over oceans [72].

Over vegetation, however, the situation is different. While even large, absorbing cloud droplets increase the 2.13 µm reflectance over dark oceans, they reduce the 2.13 µm reflectance over vegetation, which is highly reflective at this wavelength. Naturally, the reduction is stronger for thicker clouds, but, as Figure 2 shows, it levels off for very thick clouds. This makes the nonlinearity effect shift the other way. If a sample pixel contains an equal mix of a clear region and a cloud that contains large droplets with r_e = 30 µm (reflectances identified by points A and D, respectively), the pixel’s average 0.65 µm and 2.13 µm reflectances (at point E) will yield an underestimated effective radius value of around 19 µm. This implies that, over vegetated land, cloud heterogeneity can cause systematic r_e underestimations for clouds containing large droplets. Moreover, Figure 2f shows that, for very oblique sun, underestimation is expected, even for clouds containing small droplets. As cloud reflection is concentrated into forward scattering directions, the nadir 2.13 µm reflectance of the bright vegetation is reduced by the presence of even mildly absorbing small cloud droplets.

While bispectral retrievals based on 1D radiative transfer can provide reasonable (even if biased) results for relatively flat clouds such as stratocumulus, Figure 3 shows that, for cumulus clouds, this is often not the case. Three-dimensional cloud heterogeneity not only causes retrieval errors, but altogether prevents retrievals for a large portion of pixels. This is because retrievals are usually constrained to the 4–30 µm range regarding droplet effective radius (marked by yellow lines in Figure 3), and 3D radiative effects often move these (simulated) observations outside the range of reflectances corresponding to 4 and 30 µm droplet sizes.

It is important to point out that many of the outlying pixels are cloud-free and have too-high or too-low reflectances only because of the shadowing or side-scattering enhancement effect of nearby clouds. Nonetheless, the yellow text in the bottom-right corner of each figure panel reveals that a large fraction of cloudy pixels also do not allow 1D retrievals.

Finally, we note that the relatively mild curvature of the 8 µm lines in Figure 3a,b shows that the retrieval biases due to subpixel variability (discussed for Figure 2) are fairly small for the 8 µm droplet size used in this study, while Figure 3c shows that this is not the case for very oblique sun (Θ₀ = 75°). This means that, for not-too-oblique sun (Θ₀ = 15°, 45°), most errors in the r_e values provided by the 1D bispectral retrievals come from 3D radiative effects, and not from subpixel heterogeneity.

3. Proposed Technique

As discussed in the introduction, the primary goal of this study is to explore the potential help a new technique may offer in improving the current estimates of large-scale cloud statistics—namely, the mean and standard deviation of the cloud optical thickness and effective radius over areas dozens of kilometers on each side. This could help to remove systematic biases from cloud studies and improve the accuracy of Level 3 cloud products based on bispectral retrievals (e.g., MODIS or VIIRS products).

Specifically, we explore reversing the usual data processing flow, which estimates the cloud properties for each pixel and then compiles large-area statistics based on the individual pixel results. Instead, the proposed approach first determines the effect of 3D heterogeneity on 1D estimates of large-area statistics and then distributes this effect among individual pixels. The main advantage of this top-down approach is a better-posed inversion problem. It promises improved cloud statistics without requiring us to untangle the complex web of 3D radiative interactions for individual pixels.

For an initial exploration of the top-down approach, this study used a simple regression-based method to estimate the 3D effects based on readily observable scene characteristics. We considered 30 such potentially relevant scene characteristics: (i) cloud fraction; (ii) the mean and standard deviation of the cloud optical thickness and the ratio of these two; (iii) the χ-parameter defined as $\chi =e^{\overline{ln\tau }}/\overline{\tau }$, with overbars indicating averaging over all cloudy pixels [73,74]; (iv) the mean differences in along-sun or cross-sun directions over distances of one, two, or three pixels in cloudy pixels with a 0.65 µm reflectance, optical thickness, or effective cloud top height (based on brightness temperature); (v) the mean relative difference in the 0.65 µm reflectance or optical thickness of cloudy pixels neighboring each other in the along-sun direction; (vi) the standard deviation of the difference in the 0.65 µm reflectance or optical thickness of cloudy pixels neighboring each other in the along-sun direction; (vii) the ratio of the mean differences between the 0.65 µm reflectances of pixels separated by one pixel and three pixels in the cross-sun direction; and (viii) the difference between the mean reflectance, optical thickness, or effective radius over slopes facing toward and away from the sun (as identified by brightness temperature gradients) [28]. Admittedly, this was an arbitrary set chosen based on intuition to capture and take advantage of key aspects of cloud variability such as scale invariance [75]; future studies may very well identify more helpful characteristics.

We note that, in calculating these parameters, we considered pixels to be cloudy if either their true or 1D-retrieved τ-value exceeded 0.4, which is near the cloud detection sensitivity in MODIS cloud products [76]. We also note that, in the future processing of actual satellite images, one may skip an area and jump to the next one if the area is cloud-free or otherwise unsuitable for the proposed method, for example, if cloud height retrievals indicate high (ice) clouds.

Next, we tested which subset of characteristics allowed us to best estimate 3D effects using the algorithm described below (and outlined in Figure 4). These tests compared the results from using all possible combinations of up to 4 characteristics (out of the 30), and then checked whether adding any 5th characteristic to the best 4 would help. We found that, in all cases, four characteristics were sufficient, and including additional ones did not significantly improve the results. However, the exact set of the four most helpful characteristics varied with solar elevation and was different for estimations of cloud optical thickness and particle size, and even for estimations of mean or standard deviation values. Oftentimes, though, the accuracy differences between the most helpful set and a few other sets of four characteristics were minimal. As an example,

Table 1. List of scene characteristics most helpful in estimating 3D effects on $\overline{\tau }$. [Notation: Θ₀: solar zenith angle, σ: standard deviation, Δ: difference, z: cloud top height, τ: cloud optical thickness, overbar: averaging over cloudy pixels in area, as: along-sun, cs: cross-sun, numeric subscripts: distance in units of pixels, CF: cloud fraction, and A: asymmetry between cloud slopes facing toward and away from the sun].

Θ0	Most Helpful Parameters
	i	ii	iii	iv
15°	${\sigma }_{\tau }$	${\overline{∆z}}_{as,1}$	${\overline{∆\tau }}_{cs,1}$	CF
45°	${\sigma }_{\tau }$	${\overline{∆z}}_{as,1}$	${\overline{∆\tau }}_{cs,1}$	${\overline{∆z}}_{as,2}$
75°	${\sigma }_{\tau }$	${\overline{∆z}}_{as,1}$	${\overline{∆\tau }}_{as,3}$	Aτ

lists the characteristics that were most helpful for estimating the influence of 3D heterogeneity on the scene-averaged values of 1D-retrieved cloud optical thickness ($\overline{\tau }$) values. The table reveals that, as the sun becomes more oblique and photons travel larger horizontal distances, it becomes increasingly helpful to consider pixel-to-pixel differences over larger distances in the along-sun direction. For very oblique sun, even the asymmetry (A) between the mean τ-values retrieved over slopes facing toward and away from the sun becomes quite helpful.

Step 1 of the proposed top-down technique estimates—given a set of N_c number of characteristics out the 30 described above (N_c ≤ 5)—the sought-after large-scale statistics such as the average cloud optical thickness of (20 km)² scenes. This involves two sub-steps.

First, a preliminary estimate (${\overline{\tau }}_p$) is obtained through linear regression using,

$${\overline{\tau }}_p={\overline{\tau }}_{1\mathrm{D}}+∆{\overline{\tau }}_p={\overline{\tau }}_{1\mathrm{D}}+\left[a+\sum _{i=1}^{N_c}{b}_i{x}_i\right]$$

(3)

where ${\overline{\tau }}_{1\mathrm{D}}$ is the scene’s average τ provided by 1D retrievals, $∆{\overline{\tau }}_p$ is our preliminary adjustment for 3D effects, $x_i$ is the ith observed scene characteristic (one of the 30 mentioned above), and a and $b_i$ are empirically determined regression coefficients.

Subsequently, a final estimate (${\overline{\tau }}_f$) is obtained by another regression that can capture some nonlinearities by using a second-order polynomial in the regression,

$${\overline{\tau }}_f={\overline{\tau }}_{1\mathrm{D}}+∆{\overline{\tau }}_f={\overline{\tau }}_{1\mathrm{D}}+\left[c+d∆{\overline{\tau }}_p+e∆{\overline{\tau }}_p^2\right]$$

(4)

where $∆{\overline{\tau }}_f$ is our final adjustment for 3D effects, and c, $d$, and $e$ are empirically determined regression coefficients. (All regression coefficients are determined by minimizing the root-mean-square (RMS) difference between the true LES $\overline{\tau }$ values and the estimates from Equation (3) or (4)).

Step 2 of the proposed top-down technique takes the area-averaged 3D effect ($∆{\overline{\tau }}_f$) and distributes it among individual pixels. In practice, this is performed in two sub-steps.

The first sub-step obtains pixel-by-pixel estimates of each pixel’s τ (and Δτ), similarly to Equations (3) and (4), but without the area averaging indicated by overbars, and using pixel-specific local (and not scene-averaged) reflectance, τ, or altitude gradients and variabilities as input characteristics.

The second sub-step normalizes these pixel-to-pixel estimates so their average over the (20 km)² area matches the result we obtained in Step 1 using Equation (4). We found that the optimal method for this normalization (i.e., rescaling or shifting) depends on solar elevation; the best normalized τ-values (${\tau }_f^{*}$) for individual pixels come from,

$${\tau }_f^{*}={\tau }_{1\mathrm{D}}+{∆\tau }_f^{*}={\tau }_{1\mathrm{D}}+{∆\tau }_f{\displaystyle \raisebox{1ex}{$∆{\overline{\tau }}_f$}\!\left/ \!\raisebox{-1ex}{$\overline{∆{\tau }_f}$}\right.}\mathrm{for}{\mathsf{\Theta }}_0\le 45°$$

(5)

and from,

$${\tau }_f^{*}={\tau }_{1\mathrm{D}}+{∆\tau }_f^{*}={\tau }_{1\mathrm{D}}+{∆\tau }_f+\left[∆{\overline{\tau }}_f-\overline{∆{\tau }_f}\right]\mathrm{for}{\mathsf{\Theta }}_0=75°$$

(6)

In these equations, ${∆\tau }_f$ is the pre-normalization estimate of the individual pixel 3D effect, $\overline{∆{\tau }_f}$ is the area average of these pre-normalization individual pixel estimates (i.e., ${∆\tau }_f$ values), and $∆{\overline{\tau }}_f$ is the area-averaged 3D effect provided by Equation (4).

We note that we prefer using ${\overline{\tau }}_{1\mathrm{D}}$ as a starting point in the estimation process (instead of estimating ${\overline{\tau }}_p$ directly, without involving ${\overline{\tau }}_{1\mathrm{D}}$) because the calculation of ${\overline{\tau }}_{1\mathrm{D}}$ using an existing data-processing algorithm can take care of complicating issues such as Earth curvature, gaseous absorption, or instrument spectral response function, and not dealing with these issues can make the 3D adjustment much simpler. However, relying on the result of 1D retrievals raises the question of how to deal with 3D effects moving pixels outside the range of reflectances that allow valid 1D retrievals (i.e., outside the 1D retrieval space bounded by the $r_e$ = 4 µm and $r_e$ = 30 µm lines in Figure 3). In such cases, we force 1D retrievals to yield $r_e$ = 4 µm or $r_e$ = 30 µm and the τ-value corresponding to the “observed” 0.65 µm reflectance along the $r_e$ = 4 µm or $r_e$ = 30 µm lines in Figure 3. If the “observed” 0.65 µm reflectance lies outside the range of 1D reflectances for 0 < τ < 150, ${\tau }_{1\mathrm{D}}$ is set to 0 (cloud free) or 150.

Finally, we also note that our technique works the same way for estimating r_e instead of τ, and for estimating standard deviation instead of mean value—but for brevity, we do not include the equations for those estimations here.

4. Results

For an initial exploration of the technique described above, we divided our LES dataset (described in Section 2) into two parts: 70% of the scenes were used to determine the regression coefficients in Equations (3) and (4), and the remaining 30% of scenes were used to test the performance of the method. Accordingly, each dot in Figure 5, Figure 6, Figure 7 and Figure 8 will represent one of the scenes in the 30% used for testing; 396 testing scenes were used for ${\mathsf{\Theta }}_0$ = 15° and 75°, and 1049 testing scenes were used for ${\mathsf{\Theta }}_0$= 45°.

Figure 5 shows that the proposed method was effective in removing systematic biases from the scene-averaged cloud optical thickness values estimated via traditional 1D retrievals.

Figure 6 shows that the proposed method also had considerable skill in estimating the error of 1D retrievals—that is, the adjustment needed to correct the scene-averaged cloud optical thickness values. Panels (a) and (b) show that, for moderate sun angles, the method captured and correctly accounted for over 80% of the scene-to-scene variability in 1D retrieval errors (R² > 0.8) and reduced 4-fold the mean average errors (MAEs) to a level below 0.7. Panel c shows that, at very oblique sun, the error reduction was smaller, but the method yielded even more accurate estimates (smaller MAE) than for less oblique sun.

We note that, while the proposed method fully eliminates systematic biases for this dataset, it may not perform quite so well for other datasets that include statistically different cloud structures. Even so, the method can be expected to reduce the overall biases to somewhere between a perfect reduction to zero and a reduction by the same factor that MAE is reduced by in this figure. (This assumes that the distributions of cloud variability in a future dataset and the current dataset will not differ from each other by more than the distributions of cloud variability in individual scenes and in the dataset as a whole tend to differ from each other—or that, if this assumption does not hold, the regression coefficients will be recalculated using an improved dataset.)

To put the results in Figure 6 in perspective, it is helpful to test how the accuracy of the proposed top-down approach compares to that of the traditional bottom-up approach, which first estimates the 1D retrieval errors for each 1 km size pixel and then averages the results over large areas. For these comparisons, we used the same regression-based technique in both approaches: For the top-down approach, we used Equations (3) and (4), and for the bottom-up approach, we the same equations, but without the overbars and using local (as opposed to scene-wide) gradients and variabilities as inputs.

Table 2. Comparison of two approaches for estimating the error of scene-averaged cloud optical thickness (τ) values provided by 1D retrievals: (1) the traditional bottom-up approach of averaging 1 km resolution pixel-by-pixel error estimations over each scene and (2) the proposed top-down approach of estimating scene-average errors based on scene variability statistics. [Notation: Θ₀: solar zenith angle, R²: coefficient of determination (square of the correlation between true and estimated errors of 1D retrievals), and MAE: mean absolute error].

Θ0	Bottom-Up Approach		Top-Down Approach
	R2	MAE	R2	MAE
15°	0.53	0.99	0.81	0.65
45°	0.58	0.85	0.85	0.53
75°	0.10	0.63	0.43	0.43

shows that, given similar inputs and (regression-based) methodology, the proposed top-down approach outperforms the traditional bottom-up approach. Its higher R² values show that it can explain a larger fraction of variability in the 1D retrieval errors that we tried to correct, and its lower MAE values show that it yields more accurate results.

As mentioned earlier, most calculations in this paper considered only pixels where cloud detection could be expected if actual (and not simulated) satellite observations were used (i.e., ${\tau }_{1\mathrm{D}}$ > 0.4). However,

Table 3. Accuracy in estimating the impact of 3D effects on the average optical thickness $\overline{\tau }$ only for cloudy pixels with ${\tau }_{1\mathrm{D}}$ > 0.4 (where clouds are likely to be detected by instruments such as MODIS), or for entire scenes (including even the thin-cloud or cloud-free pixels where instruments such as MODIS are unlikely to detect any cloud presence, thus capturing the effect of 3D heterogeneities even over undetected clouds). Higher R² values and smaller MAE/MAE_1D ratios indicate better performance. [Notation: Θ₀: solar zenith angle, R²: coefficient of determination (square of the correlation between the true and estimated errors of 1D retrievals), and MAE: mean absolute error. No subscript means the values are for the proposed method, while the subscript “1D” identifies the MAE of assuming that 1D retrievals provide accurate results].

Θ0	${\mathit{\tau }}_{1\mathbf{D}}$ > 0.4 Only		Entire Scene
	R2	MAE/MAE1D	R2	MAE/MAE1D
15°	0.81	0.25	0.88	0.33
45°	0.85	0.24	0.83	0.41
75°	0.43	0.64	0.69	0.20

shows that the proposed method had comparable skill in obtaining the average optical thickness of all clouds, even including the pixels where instruments such as MODIS would not detect a cloud. This suggests that the approach could also help researchers to consider the statistical impacts of thin, small, or shadowed clouds that tend to be missed by cloud detection algorithms.

After considering the scene-averaged τ-values, we can also check whether the proposed approach could also help in improving characterizing the τ variability within each scene. Specifically, Figure 7 demonstrates that the proposed approach can significantly improve our estimates not only for mean cloud properties, but for cloud variability as well. Namely, the proposed approach reduces by a factor from two to four the errors that 3D effects cause in the estimated standard deviation of the cloud optical thicknesses encountered within a scene. We note that this standard deviation is part of some Level 3 satellite data products such as the MODIS or VIIRS cloud products [77,78].

Having explored the potential for improving optical thickness statistics, Figure 8 examines the potential for improving the retrieved mean values of droplet effective radii, $r_e$. (Standard deviation of $r_e$ values within each scene is not examined, as our dataset contains constant $r_e$ = 8 µm throughout all scenes, implying $\sigma \left(r_e\right)=0$, which would provide a lopsided view revealing only increases but no decreases in $\sigma \left(r_e\right)$). The figure shows that the proposed approach can significantly reduce the current retrieval biases not only for optical thickness, but for cloud droplet size as well. It captures 65–90% of the scene-by-scene variability in the 3D effects on the 1D $r_e$ retrievals and reduces the mean average error to less than 0.6 µm.

The results, so far, point toward a potentially useful skill in improving the retrievals of cloud properties based on 1 km resolution observations representative of data from polar-orbiting instruments such as MODIS or VIIRS. Thus, it can be of interest to test whether the proposed approach could also help in interpreting coarser-resolution observations, such as those of the Advanced Baseline Imager (ABI) on GOES 16 and 17, the Advanced Himawari Imager (AHI) on Himawari-8, or the Flexible Combined Imager (FCI) on Meteosat-12. To check this, we degraded the 1 km resolution simulated reflectance fields to 2 km and 4 km resolutions and re-tested the performance of the proposed approach for the coarse-resolution data.

Table 4. Accuracy statistics for estimating the errors of the $\overline{\tau }$ scene-averaged cloud optical thickness values provided by 1D retrievals. [Notation: Θ₀: solar zenith angle, R²: coefficient of determination (square of the correlation between true and estimated errors), and MAE: mean absolute error. No subscript means the values are for the proposed method, while the subscript “1D” identifies the MAE or bias of assuming that 1D retrievals provide accurate results].

Resolution [km]	Θ0	$\overline{\mathit{\tau }}$
		R2	MAE	Bias	MAE1D	Bias1D	True Mean $\overline{\mathit{\tau }}$
1	15°	0.81	0.65	0.0	2.61	−2.61	4.65
	45°	0.85	0.53	0.0	2.17	−2.16
	75°	0.43	0.43	0.0	0.67	0.38
2	15°	0.74	0.53	0.0	1.70	−1.70	3.23
	45°	0.81	0.57	0.0	1.76	−1.75
	75°	0.38	0.43	0.0	0.59	−0.42
4	15°	0.71	0.53	0.0	1.46	−1.46	2.74
	45°	0.77	0.53	0.0	1.38	−1.36
	75°	0.63	0.51	0.0	0.92	−0.85

,

Table 5. Same as Table 4, but for estimating the error in the standard deviation (and not the mean) of cloud optical thickness values within each scene (σ_τ).

Resolution [km]	Θ0	στ
		R2	MAE	Bias	MAE1D	Bias1D	True Mean στ
1	15°	0.84	1.18	0.0	4.70	−4.70	6.24
	45°	0.81	1.14	0.0	4.56	−4.56
	75°	0.66	0.92	0.0	2.06	1.53
2	15°	0.74	0.71	0.0	2.17	−2.17	3.39
	45°	0.79	0.82	0.0	2.64	−2.64
	75°	0.74	0.59	0.0	1.11	−0.17
4	15°	0.75	0.51	0.0	1.27	−1.27	1.87
	45°	0.70	0.53	0.0	1.32	−1.32
	75°	0.70	0.45	0.0	0.78	−0.61

and

Table 6. Same as Table 4, but for the scene average droplet effective radius ${\overline{r}}_e$ instead of the scene average cloud optical thickness $\overline{\tau }$.

Resolution [km]	Θ0	${\overline{\mathit{r}}}_{\mathit{e}}$
		R2	MAE	Bias	MAE1D	Bias1D	True Mean ${\overline{\mathit{r}}}_{\mathit{e}}$
1	15°	0.70	0.58	0.0	3.52	3.52	8.00
	45°	0.65	0.56	0.0	0.91	−0.08
	75°	0.90	0.29	0.0	2.52	−2.46
2	15°	0.47	0.62	0.0	2.78	2.77	8.00
	45°	0.53	1.15	0.0	4.72	4.69
	75°	0.86	0.40	0.0	2.32	−2.14
4	15°	0.26	0.55	0.0	2.34	2.34	8.00
	45°	0.51	2.10	0.0	12.3	12.3
	75°	0.83	0.50	0.0	2.56	−2.23

suggest that, while the new method worked best for 1 km resolution satellite data (which can provide detailed statistical information of cloud variability), the proposed method can bring about significant improvements even for satellite data with a coarser (2 km or even 4 km) resolution. All biases were eliminated by design, R² values dropped only modestly (${\overline{r}}_e$ at high sun was the one exception), and all MAE values remained significantly lower for the proposed method than for the 1D retrievals.

We note that one factor that helped the MAE values to remain small or even drop for coarser resolutions in Table 4 and Table 5 is that, during spatial averaging, error increases due to losses of information were counteracted by drops in the true $\overline{\tau }$ and σ_τ values of cloudy pixels. The true mean τ values decrease for coarser resolutions in Table 4 because spatial averaging increases cloud cover. In calculating the average τ of cloudy pixels at a coarser resolution, we divide the unchanged amount of scatterers by a larger cloudy area. (The cloudy area increases because the entire area of a coarse-resolution pixel is considered as cloudy, even if clouds cover only a part of the pixel.) The true σ_τ values decrease for coarser resolutions in Table 5 both because the τ values decrease in general (as mentioned above) and because averaging washes out small-scale variability.

Although the main purpose of the proposed approach is to improve large-scale cloud statistics, the final test of this study explores the second step of the proposed approach: distributing the estimated area-averaged 3D effects among individual pixels. Figure 9 illustrates that, while the new method offers substantial improvements for estimating large-area cloud statistics (as shown in preceding figures and tables), it provides more modest improvements for estimating the cloud properties at individual pixels. This is precisely the reason that motivated the development of the new top-down approach. Correctly estimating radiative heterogeneity effects is a more difficult endeavor for individual pixels than for large areas. We emphasize, however, that further improvements are not at all out of the question—and that even as is, the method removes most systematic biases from the individual pixel estimates and (as indicated by R² ≈ 0.5) correctly accounts for about half of the variability in the errors of the 1D τ retrievals.

5. Summary and Conclusions

This paper presented a study that was motivated by two key points, as follows: (i) satellite measurements of cloud optical properties are critically important for reducing key uncertainties in climate prediction and (ii) current methods for such satellite measurements use the one-dimensional (1D) approximation of horizontal homogeneity, which causes errors for both individual pixels and large-scale cloud statistics. The specific goal was to explore a new approach to improve the satellite measurements of cloud optical depth and droplet size by considering the radiative impacts of horizontal heterogeneity. The main focus was on improving large area statistics, such as Level 3 satellite products, although the study also briefly explored the issue of improving the accuracy for individual pixels in satellite images.

To improve large-area cloud statistics, this study explored a new approach to the remote sensing of heterogeneous clouds. While past efforts have used a bottom-up approach that starts with retrievals for individual pixels and then subsequently compiles large-scale statistics, this study reversed the order. We sought to first determine the effect of 3D heterogeneity on 1D estimates of large-scale cloud statistics and then distribute the overall effects to individual pixels. The main advantage of this top-down approach is a better-posed inversion problem. We can seek accurate statistics without untangling the complex web of 3D radiative interactions that has, for decades, stymied the retrievals of cloud properties for individual pixels in heterogeneous scenes.

Although this study only considered adjustments to bispectral retrievals based on a single instrument (not unlike MODIS or VIIRS or geostationary satellite imagers), the top-down approach (in which the key inputs are statistical parameters) may also be easier to apply in situations where data are combined from sensors with different view directions, spatial resolutions, or observation times.

To explore this top-down approach, we implemented it in a simple algorithm and tested its performance using a dataset containing over 3000 scenes of boundary-layer cumulus clouds generated through large eddy simulations (LESs). The dataset included both 3D cloud structures and the corresponding radiation fields simulated by a 3D Monte Carlo radiative transfer model. These radiation fields consisted of 0.65 µm, 2.13 µm, and 11 µm radiances that a 1 km resolution satellite looking straight down would observe.

Using this dataset, we tested the accuracy of our simple regression-based algorithm that uses statistical properties such as mean horizontal gradients to adjust the scene-averaged cloud optical thickness and effective radius values provided by 1D retrievals. The main results from these tests were as follows.

• The results showed that the algorithm greatly reduced the heterogeneity-caused biases in 1 km resolution satellite retrievals of cloud optical thickness and effective radius. For individual (20 km)² cloud fields, the algorithm reduced the typical mean errors of 1D retrievals to below 0.7 in optical thickness and 0.6 µm in effective radius. In most cases, this meant a reduction by a factor of four or more. Overall biases and errors for larger individual areas are expected to drop even further and be close to zero.
• The results indicated that, in addition to removing biases from scene-averaged cloud parameters, the new algorithm also improved our estimates of the cloud variability within each cloud field. Specifically, it significantly reduced (by a factor ranging from two to four) the errors that 3D radiative effects cause in current estimates of the standard deviations of cloud optical thicknesses. We note that this standard deviation is also part of some (e.g., MODIS) Level 3 satellite data products.
• The data revealed that, while the algorithm worked best for 1 km resolution satellite data, it brought about almost as large improvements for 2 km resolution satellite images. Even for 4 km resolution images, the algorithm reduced current errors by factors ranging from about two to six. This implies that the algorithm can help to improve even long-term time series that include data from past or future coarse-resolution (e.g., geostationary) satellites.
• The results indicated that the proposed top-down approach provided more accurate cloud statistics than comparable bottom-up estimations of large-scale statistics based on individual pixel retrievals.
• The results also yielded scientific insights into the impact of cloud heterogeneities in satellite remote sensing. Most importantly, they revealed that 1D bispectral satellite retrievals of cloud droplet size often have biases with opposite signs over vegetated surfaces and over oceanic areas. While nonlinearities tend to result in overestimations of cloud droplet sizes over oceans, they cause underestimations over vegetation if the cloud droplet effective radius is large (≥15 µm) or the sun is low above the horizon.

The current study sought only to provide initial insights about the potential of a proposed approach, and it is very important to recognize its limitations. These limitations need to be overcome if this approach is pursued in future studies or perhaps even operational applications. First, future studies should consider a wider variety of scene and observation parameters, for example, by including more view directions and surface albedos, 3D variations in droplet effective radius, ice crystals in addition to liquid water droplets, and scenes from several LES models and atmospheric conditions—including larger scenes that, for simulations over water, should include cloud organization patterns other than the “sugar” pattern used here. Due to the large computational needs of 3D radiative transfer simulations, such expansions in scene and observation parameters will likely need to rely heavily on interpolation (e.g., in view direction or surface albedo), and perhaps take advantage of emerging emulators for quick estimations of the radiation fields in heterogeneous cloud scenes [79]. Real-world applications would also need to develop criteria for identifying the scenes where the proposed approach can be applied. Establishing such criteria may involve a satellite-level verification of any improvement in retrieval capabilities, which is not yet available for the proposed method and some other promising techniques such as neural networks. Finally, future studies may seek improvements to the regression-based estimations of scene statistical parameters, perhaps by using more complex nonlinear regressions based on physical insights or additional input scene characteristics, and perhaps by using machine learning techniques other than the straightforward regressions used here.

Overall, our main conclusion from this study is that the proposed top-down approach does indeed show potential to improve the accuracy of both old and new satellite datasets.