Enhanced Resolution of Microwave Sounder Imagery through Fusion with Infrared Sensor Data

Abstract

The images acquired by microwave sensors are blurry and have low resolution. On the other hand, the images obtained using infrared/visible sensors are often of higher resolution. In this paper, we develop a data fusion methodology and apply it to enhance the resolution of a microwave image using the data from a collocated infrared/visible sensor. Such an approach takes advantage of the spatial resolution of the infrared instrument and the sensing accuracy of the microwave instrument. The model leverages sparsity in signals and is based on current research in sparse optimization and compressed sensing. We tested our method using a precipitation scene captured with the Advanced Microwave Sounding Unit (AMSU-B) microwave instrument and the Advanced Very High Resolution Radiometer (AVHRR) infrared instrument and compared the results to simultaneous radar observations. We show that the data fusion product is better than the original AMSU-B and AVHRR observations across all statistical indicators.

1. Introduction

Microwave sensors are able to penetrate through thick clouds to see the structure of a storm. The images collected are valuable for evaluating the storm’s internal processes and its strength. However, the data (e.g., brightness temperatures) acquired by microwave (MW) sensors are blurry and of low-resolution, and thus all derived products, including rain rates will share that characteristic. On the other hand, the images obtained using infrared/visible sensors (IR/Vis), and their corresponding products can offer higher resolution but with negligible ability to penetrate into clouds. In some cases, we can use the data from a collocated infrared/visible sensor to increase the resolution of a microwave image.

We considered the deconvolution inverse problem in [1], where we deblurred images from the effects of a point spread function. Additionally, in [1], we developed a simultaneous deconvolution and upsampling algorithm in order to enhance the effective spatial resolution of an image. Since the convolution problem in the presence of noise is highly ill-posed, regularization was applied to achieve stability while preserving a priori properties of the solution. We formulated the restoration problem within the variational framework, using the total variation regularization [2]. Total variation (TV) of an image measures the sum of the absolute values of its gradient and increases in the presence of irregularities. By minimizing the TV, we showed that the process significantly reduces the brightness temperature errors in the overall image. These processes were rendered efficiently by employing methodologies based on current research in sparse optimization and compressed sensing. We performed the total variation-based deconvolution within the split Bregman optimization framework to achieve a significant computational time improvement over already robust total-variation gradient descent-based techniques.

The split Bregman method can be derived from the well-known alternating direction method of multipliers (ADMM) [3,4,5], and is very efficient because it can decompose a non-smooth multi-term optimization problem into subproblems with closed-form solutions. This advantage of total variation regularization was first discovered in [6] for image denoising and deblurring, and has been generalized to multichannel problems in [7], the TV-L1 model in [8], TV-based compressed sensing in [9,10], and an edge-guided compressive sensing reconstruction approach for recovering images of higher qualities from fewer measurements in [11].

The fusion of sensor data is a well-established phenomenon, which appears in many different applications. In [12], the authors investigated the multisensor data fusion task. The paper proposed a comprehensive review of the data fusion state of the art, exploring its conceptualizations, benefits, and challenging aspects, as well as existing methodologies. In [13], the existing literature on multimodal fusion research is presented through several classifications based on the fusion methodology and the level of fusion. In [14], a comprehensive survey of the mixture of experts is provided. The authors discuss the fundamental models for regression and classification and also their training with the expectation-maximization algorithm. In [15], the problem of combining classifiers which use different representations of the patterns to be classified was studied. The problem of optimum decision fusion in the presence of statistical dependence among the individual decisions was considered in [16]. The authors in [17] investigated a fuzzy multi-sensor data fusion Kalman filter model for reducing risk in an integrated vehicle health maintenance system. Berliner et al. [18] introduced a hierarchical Bayesian spatio-temporal model with multiresolution wavelet basis functions and two data sources of different support. Wikle et al. [19] developed a hierarchical Bayesian spatio-temporal dynamic model that combines wind data from different sources with background physics to produce realizations of high-resolution surface wind fields. Banerjee et al. [20] proposed a class of models that is based on the idea of a spatial predictive process, which is motivated from kriging ideas. Their approach is in the same spirit as process model approaches using basis functions and kernel convolutions. Cressie and Johannesson [21] introduced fixed rank kriging based on the spatial random effects model, and Cressie et al. [22] introduced fixed rank filtering and fixed ranked smoothing based on the Kalman filter and the spatio-temporal random effects model. Nguyen et al. [23] introduced a spatio-temporal data fusion methodology based on reduced-dimensional Kalman smoothing. Their method is able to combine data captured by two instruments using a Bayesian framework.

In this paper, we consider the simultaneous data fusion, deconvolution, and upsampling problem, where we not only enhance the effective resolution of a microwave image, but further enhance its resolution using the data from an infrared instrument. Such an approach takes advantage of the high spatial resolution of the infrared instrument and the sensing accuracy of the microwave instrument. In our experiments, we used data captured with the infrared Advanced Very High Resolution Radiometer (AVHRR) instrument and the Advanced Microwave Sounding Unit (AMSU-B) microwave instrument, both flying on the NOAA-18 satellite. We tested our method using precipitation scenes captured with AMSU-B and AVHRR both over the ocean and ground, and compared the results to ground radar observations. The purpose of the paper is to provide a preliminary report on methodology. Because ground radar observations are difficult to acquire, only two events are considered in the paper. We apply our novel method to the existing available data.

2. Materials and Methods

2.1. Mathematical Notation

We first introduce notations that will be used throughout this paper. For an image $u\in {\mathbb{R}}^{n\times n}$, the value of u at a pixel $(i,j)$, with $0\le i,j\le n$, is denoted as $u_{ij}$. The norms are defined as:

$${\left|\right|u\left|\right|}_1=\sum _{(i,j)\in \mathsf{\Omega }}|u_{ij}{|,|\left|u\right||}_2=\sqrt{\sum _{(i,j)\in \mathsf{\Omega }}{\left|u_{ij}\right|}^2}.$$

The gradient of u is denoted as $\nabla u$, and its value at pixel $(i,j)$ as $\nabla u_{ij}$, with $\nabla u_{ij}\in {\mathbb{R}}^2$. For a vector-valued quantity $d_{ij}=({\left(d_1\right)}_{ij},{\left(d_2\right)}_{ij})\in {\mathbb{R}}^2$ (e.g., $d=\nabla u$), the norms are defined as

$${\left|\right|d\left|\right|}_1=\sum _{(i,j)\in \mathsf{\Omega }}\left|\right|d_{ij}{\left|\right|}_2{,\left|\right|d\left|\right|}_2=\sqrt{\sum _{(i,j)\in \mathsf{\Omega }}\left|\right|d_{ij}{\left|\right|}_2^2},$$

where $\left|\right|d_{ij}{\left|\right|}_2=\sqrt{{\left(d_1\right)}_{ij}^2+{\left(d_2\right)}_{ij}^2}$. Unless specified otherwise, $\left|\right|·\left|\right|=\left|\right|·{\left|\right|}_2$ in the remainder of this paper.

2.2. Data Fusion

Microwave images are inherently of low spatial resolution compared with optical sensors with similar receiving apertures. The pixel size for the Advanced Microwave Sounding Unit-B (AMSU-B) microwave sensor is 15 km at nadir. Thus, for example, a 3000 by 3000 km scene can be represented on a 200 × 200 grid. This limits the scientific analysis of derived data products. Our objective is to simultaneously (i) increase the effective resolution of the observed image, upsampling the resolution by some factor in each dimension, (ii) reduce the effects of noise, (iii) preserve the edges and other features in the image, and (iv) perform fusion of infrared and microwave data in order to increase the resolution of a microwave image.

We first formulate the microwave forward imaging model, which convolves, adds noise to, and downsamples the physical scene as it is being captured by the microwave sensor. That is, the physical scene captured by the microwave sensor is defined at the lower resolution, or at the resolution of the microwave sensor. We assume that the scene has been captured at this lower resolution as a result of applying the downsampling to some unknown high-resolution scene. In order to recover the scene at the higher resolution, we will upsample the scene by inverting the downsampling problem. This would increase the effective resolution of the observed image, which is objective (i) of our method.

The downsampling process is defined by the downsampling operator. Such an operator is a transformation from a fine (high-resolution) grid to a coarse (low-resolution) grid. We denote the downsampling matrix as $D\in {\mathbb{R}}^{n\times p}$ with $p=n/k$, where k is the downsampling (or upsampling) factor. The larger the downsampling factor k is, the coarser the resulting grid would be. We assume that a physical scene $u_0$, when being captured, is convolved with an antenna kernel K, corrupted with instrument noise ${\kappa }_1$, and then downsampled with an operator D, arriving at the microwave observation $f_1\in {\mathbb{R}}^{p\times p}$:

$$f_1=D^T(K\ast u_0+{\kappa }_1)D,$$

(1)

where ∗ denotes convolution.

We note that the assumption that a physical scene is convolved with a non-trivial antenna kernel is not always the case, in which case we can assume that K is a delta function, $K=\delta$, in (1).

We assume the sensor model for the infrared sensor satisfies the additive noise model (without convolution):

$$f_2=u_0+{\kappa }_2.$$

(2)

Hence, Equations (1) and (2) form a pair of equations for the microwave and infrared rain rates $f_1$ and $f_2$, respectively. To recover an approximation to the true rain rate $u_0$, we solve the inverse problem.

In [1], by formally solving the deconvolution inverse problem, we sharpened the image while preserving information content. We used the $L_1$-regularized type norm ${\left|\right|u\left|\right|}_{TV}=\int |\nabla u|={\left|\right|\nabla u\left|\right|}_1$, which measures the total variation (TV). The minimization of the TV norm does not penalize edges in an image. In this paper, we solve the simultaneous data fusion, deconvolution, and upsampling problem within an efficient multispectral total variation-based split Bregman minimization framework.

Given the microwave observation $f_1$ and infrared observation $f_2$, we consider the following multisensor data fusion minimization problem:

$$\underset{u}{min}\left\{{\left|\right|\nabla u\left|\right|}_1+\frac{{\mu }_1}{2}\left|\right|f_1-D^T{(K\ast u)D\left|\right|}_2^2+\frac{{\mu }_2}{2}\left|\right|f_2-u{\left|\right|}_2^2\right\}.$$

(3)

Non-negative weights ${\mu }_1$ and ${\mu }_2$, which may be spatially variant, indicate the plausibility of microwave and infrared rain rates, respectively. Note that the second and third terms in Equation (3) represent residuals of Equations (1) and (2), respectively. The downsampling operator D is modeled to transform the image from a finer IR grid to a coarser MW grid. Our split Bregman-based optimization algorithm will recover the deconvolved and noiseless image u defined on a high-resolution IR grid. The process of deconvolving antenna kernel K, together with applying total variation regularization, would reduce the effects of noise and preserve the edges and other features in the image, which are objectives (ii) and (iii). The presence of the second and third terms in Equation (3) achieve fusion of infrared and microwave data in order to increase the resolution of a microwave image, which is objective (iv).

In order to minimize (3), an additional variable $d$ is introduced to transfer $\nabla u$ out of non-differentiable terms at each pixel, and $\left|\right|d-{\nabla u\left|\right|}^2$ is penalized. Hence, the split Bregman formulation of problem (3) is

$$\begin{array}{c}\hfill \underset{u,d}{min}{\left|\right|d\left|\right|}_1+\frac{\lambda }{2}\left|\right|d-\nabla u-{b\left|\right|}^2+\frac{{\mu }_1}{2}\left|\right|f_1-D^T{(K\ast u)D\left|\right|}^2+\frac{{\mu }_2}{2}\left|\right|f_2-u{\left|\right|}^2.\end{array}$$

(4)

Here, $\lambda$ is a non-negative parameter, and variable $b$ is chosen through Bregman iteration [24,25]:

$$\begin{array}{ccc}\hfill b& \leftarrow & b+(\nabla u-d).\hfill \end{array}$$

(5)

For a fixed u, the minimization problem for $d$ is

$$\begin{array}{c}\hfill d^{\ast }=arg\underset{d}{min}\left\{{\left|\right|d\left|\right|}_1+\frac{\lambda }{2}\left|\right|d-\nabla u-{b\left|\right|}^2\right\},\end{array}$$

which can be explicitly solved for $d$, at each pixel, by using a generalized shrinkage formula [26,27]:

$$d=max\left\{\left|\right|\nabla u+b\left|\right|-\frac{1}{\lambda },0\right\}\frac{\nabla u+b}{\left|\right|\nabla u+b\left|\right|}.$$

(6)

For a fixed $d$, the minimization problem (4) is quadratic in u:

$$u^{\ast }=arg\underset{u}{min}\left\{\left|\right|d-\nabla u-{b\left|\right|}^2+\frac{{\mu }_1}{\lambda }\left|\right|f_1-D^T{(K\ast u)D\left|\right|}^2+\frac{{\mu }_2}{\lambda }\left|\right|f_2-u{\left|\right|}^2\right\},$$

and has the optimality condition:

$$▵u-\nabla ·(d-b)+\frac{{\mu }_1}{\lambda }\tilde{K}\ast \left(D\left(f_1-D^T(K\ast u)D\right)D^T\right)+\frac{{\mu }_2}{\lambda }(f_2-u)=0,$$

(7)

where $\tilde{K}(x,y)=K(-x,-y)$. We parametrize the descent direction by an artificial time t and solve the Euler–Lagrange equation in $u\left(t\right)$ using the gradient descent method:

$$\frac{\partial u}{\partial t}=▵u-\nabla ·(d-b)+\frac{{\mu }_1}{\lambda }\tilde{K}\ast \left(D\left(f_1-D^T(K\ast u)D\right)D^T\right)+\frac{{\mu }_2}{\lambda }(f_2-u).$$

(8)

Algorithm 1 below is a description of the developed data fusion method.

Algorithm 1: Data fusion methodology for enhancing resolution of a microwave data.

1: Initialize: Tolerance $ϵ$, $k=0$, $u^0=\mathrm{mean}\left(f_1\right)$, $d=\mathbf{0}$, $b=\mathbf{0}$. 2: Solve (6) for $d$. 3: Update $b$ as in (5). 4: Solve (8) for u. 5: Set $u^{k+1}=u$. 6: If $\left|\right|u^{k+1}-u^k\left|\right|<ϵ$, then stop. 7: Set $k:=k+1$, and go to step 2.

2.3. Data

The cross-track scanning antenna of the Advanced Microwave Sounding Unit-B (AMSU-B) microwave sensor has both a field of view (FoV) and a sampling increment of 1.1 degrees, resulting in 15 km per pixel at nadir. The same antenna is used for all five channels (89, 150, 183+/-1, 183+/-3, and 183+/-7 GHz), and aperture equalization results in the same resolution for all channels and also the retrieved MW rain rate. The Advanced Very High Resolution Radiometer (AVHRR) has a FoV and sampling increment of 0.08 degrees, resulting in ~1.1 km per pixel at nadir. The FoV is the same for all filters of the wheel (0.6, 0.8, 1.6, 3.7, 11, and 12 $\mathsf{\mu }$m), resulting in the same resolution for all channels and the retrieved IR rain rate. However, the 1.1 km resolution—called full-resolution area coverage (FRAC)—is not available on a global scale, and only a reduced dataset of $4\times 4$ km${}^2$ is available in remote areas—called global area coverage (GAC).

Collocation was done by using latitude and longitude from every pixel and finding the nearest AVHRR pixel to the center of an AMSU-B pixel. The other AVHRR pixels within the FoV of AMSU-B can then be weighted according to the distance from the center. This approach can result in uncertainties for higher scan angles, due to the change in footprint size and blurring due to the movement of the scanning instrument. The data was re-gridded on a constant grid for our purpose.

2.4. Infrared Precipitation Retrieval

IR precipitation retrieval was obtained following the three steps described below:

(1)

Collocating IR (~11 $\mathsf{\mu }$m) from the Visible and Infrared Scanner (VIRS) with precipitation rate from Precipitation Radar (PR), both aboard the Tropical Rainfall Measuring Mission (TRMM). The collocation was obtained from the University of Utah TRMM precipitation and cloud feature database [28] and used as our training datasets.

(2)

Establishing an empirical relationship between collocated IR brightness temperatures and PR precipitation estimates to map IR imagery to surface rainfall rates. This was performed using probability/histogram matching methods in which the cumulative distribution functions of precipitation rates and IR brightness temperatures are matched to provide IR-rain-rate equations under the general assumption that colder clouds statistically produce more intense rainfall [29,30,31].

(3)

The established relationship between IR brightness temperature and precipitation rate was used to retrieve precipitation intensity from AVHRR IR images, providing IR data (~11 $\mathsf{\mu }$m) similar to VIRS.

3. Results

We tested the split Bregman-based data fusion method for increasing resolution of a microwave instrument using a precipitation scene captured with AMSU-B and AVHRR instruments over the ocean (Figure 1). The captured images represent rain rates. In our experiment, the AVHRR measurements had eight times higher spatial resolution than the AMSU-B measurements. Hence, our goal was to recover a high-resolution representation of a scene on a $288\times 464$ grid (1.9 km resolution) from a low-resolution representation on a $36\times 58$ grid (15 km resolution). The algorithm generated results using these datasets in 372 s using 2.9 GHz Intel Core i7 processor.

As seen in Figure 1, the data fusion result takes advantage of the spatial resolution of the infrared instrument by defining the result on a finer grid, and also takes advantage of the sensing accuracy of the microwave instrument by representing the features of the AMSU-B rain rates in the data fusion result. The results clearly show the location of the storm eye and the eyewall around it. This is not clear in the AMSU-B image due to its relatively poor spatial resolution. This is also the case for the storm’s rain band—an area of rainfall which is largely elongated. In Figure 2, we show the energy given by Equation (3) per iteration. We see that energy decreases.

We also tested the split Bregman-based data fusion method on a continental storm over the United States and the Gulf of Mexico on 9 January 2013 for enhancing the resolution of microwave observations (Figure 3). The rain event was associated with a strong upper-level storm system moving from northern Mexico across central Texas towards the northeast. The storm collected moisture from the Gulf of Mexico and warmer air from the South, resulting in widespread rain over western Texas during landfall. Rainfall rates between 1 and 2 inches were reported all over Texas during this event. This event was captured as snapshots by AMSU-B (Figure 3a) and AVHRR (Figure 3b) instruments at 3:00 a.m. local time, taken during the overflight of NOAA 18.

Since the AVHRR measurements had eight times higher spatial resolution than the AMSU-B measurements, our goal was to recover a high-resolution image on a $712\times 1288$ grid (1.9 km resolution) from a low-resolution image on a $89\times 161$ grid (15 km resolution). Instantaneous rain rate measurements from ground weather radar were used to evaluate the recovered high-resolution image. The radar data used in this study is from the Multi-Radar/Multi-Sensor (MRMS) system, which generates automated, seamless national 3D radar mosaic and multisensory precipitation estimates at high resolution (1 km, 2 min) [32]. The radar data from the previous 24 h can be downloaded from http://mrms.ncep.noaa.gov/data. The historic archive is currently not available to the public, but will be available in the future.

As seen in Figure 3, the data fusion result takes advantage of the high spatial resolution of the infrared instrument. Compared to original AMSU-B and AVHRR observations, the data fusion product is closer to weather radar measurements in terms of storm coverage, intensity, and storm center.

Table 1. Statistical results for the Advanced Microwave Sounding Unit (AMSU-B), the Advanced Very High Resolution Radiometer (AVHRR), and the fusion product. The method with the best performance according to the statistic is denoted. Ground weather radar quantitative precipitation estimation (QPE) is used as reference. CC: correlation coefficient; MSE: mean absolute error; RMSE: root-mean-square error.

	Relative Bias (%)	CC	MAE (mm/h)	RMSE (mm/h)
AMSU-B	2.58	0.20	2.25	4.11
AVHRR	185.95	0.26	3.72	5.03
Fusion product	−1.78	0.29	1.44	3.21

shows the statistics of the original AMSU-B and AVHRR as well as the data fusion product using ground weather radar measurements as reference. The data fusion product outperforms the original microwave and IR observations across all statistical indicators, including relative bias, correlation coefficient (CC), mean absolute error (MAE), and root-mean-square error (RMSE).

Compared to AVHRR, the data fusion product mitigated the overestimation (Figure 4b). The discrepancy between data fusion product and AMSU-B exists on the edge of the storms (Figure 4a), and is not as significant as the one with AVHRR, indicating the data fusion result takes advantage of the sensing accuracy of the microwave instrument by representing the features of the AMSU-B rain rates.

4. Discussion

There remain a few concerns that we would like to communicate with the readers: (1) Microwave humidity sounders such as AMSU-B do not oversample, and they operate at higher frequencies where the signal is not fully related to surface rainfall but rather correlated to ice aloft. Given that ice aloft is not well correlated to rainfall at the surface due to shear, cloud lifecycle issues, and simple parallax (off nadir), it might be the case that even if an AMSU-B sensor with 4 km resolution existed, the rainfall derived through empirical relations would not yield superior correlations with ground-based radar than rainfall from the coarser resolution footprints currently available. In other words, one can argue that correlations can also improve due to averaging enough cloud processes that ice aloft starts to correlate more strongly with surface precipitation. (2) Undoubtedly, IR data can improve the resolution of precipitation as cloud-free scenes are not likely to rain, and one can also envision the converse where parallax effects from overlying deep cirrus or anvil clouds can shift the convection by a few kilometers from where the dense ice is found by the microwave sounder. (3) Equations (1) and (2) assume that the observed rain rates ($f_1$ and $f_2$) are the true rain rates, $u_0$, with noise and the appropriate convolution. In reality, however, the noise $\kappa$ is some random noise together with very coherent structural errors. A cirrus cloud overlying most of the scene, for instance, could not be characterized as an independent error at each pixel. Instead, it has an edge that may have a relation to the underlying precipitation field.

We note that our work is preliminary. It represents significant progress on a significant topic, namely enhancing spatial resolution of observed variables. Further work will be performed in the future. For instance, we plan to assess several other coincident satellite overpasses, such as GPM radar with AMSU and AVHRR.

5. Conclusions

This paper develops a data fusion methodology and applies it to enhance the resolution of a microwave image using the data from a collocated infrared/visible sensor. The images acquired by microwave sensors are blurry and of low-resolution. On the other hand, the images obtained using infrared/visible sensors are of higher resolution. Our approach takes advantage of the spatial resolution of the infrared instrument and the sensing accuracy of the microwave instrument. The model leverages sparsity in signals and is based on current research in sparse optimization and compressed sensing. We tested our method using a precipitation scene captured with the Advanced Microwave Sounding Unit (AMSU-B) microwave instrument and the Advanced Very High Resolution Radiometer (AVHRR) infrared instrument and compared the results to the radar observations. We showed that the data fusion product has a lower relative bias, higher correlation coefficient (CC), lower mean absolute error (MAE) and root-mean-square error (RMSE) than both AMSU-B and AVHRR observations.