Edge-Preserving Denoising of Image Sequences

Abstract

To monitor the Earth’s surface, the satellite of the NASA Landsat program provides us image sequences of any region on the Earth constantly over time. These image sequences give us a unique resource to study the Earth’s surface, changes of the Earth resource over time, and their implications in agriculture, geology, forestry, and more. Besides natural sciences, image sequences are also commonly used in functional magnetic resonance imaging (fMRI) of medical studies for understanding the functioning of brains and other organs. In practice, observed images almost always contain noise and other contaminations. For a reliable subsequent image analysis, it is important to remove such contaminations in advance. This paper focuses on image sequence denoising, which has not been well-discussed in the literature yet. To this end, an edge-preserving image denoising procedure is suggested. The suggested method is based on a jump-preserving local smoothing procedure, in which the bandwidths are chosen such that the possible spatio-temporal correlations in the observed image intensities are accommodated properly. Both theoretical arguments and numerical studies show that this method works well in the various cases considered.

1. Introduction

The Landsat project, led by the US Geological Survey (USGS) and NASA, has launched eight satellites since 1972 to continuously provide scientifically valuable images of the Earth’s surface. These images can be freely accessed by researchers around the world (cf., Zanter [1]). This rich archive of Landsat images has become a major resource for scientific research about the Earth’s surface and its resources in different scientific disciplines, including forest science, climate science, agriculture, ecology, fire science, and many more. As an example, Figure 1 shows two images of the Las Vegas area in Nevada taken in 1984 and 2007, respectively. These two images clearly show the increasing urban sprawl of Las Vegas during the 23-year period, and consequently, the environment in that region has changed dramatically. The current satellite (i.e., the Landsat 8) can deliver an image of a given region roughly every 16 days. So, we have a sequence of images of that region collected sequentially over time, stored in the Landsat database, which is increasing all the time. Image sequences are commonly used in many other applications, including functional magnetic resonance imaging (fMRI) in neuroscience and quality control in manufacturing industries (Qiu [2]). In practice, observed images usually contain noise and other contaminations (Gonzalez and Woods [3]). For reliable subsequent image analyses, such contaminations should be removed in advance. In the image processing literature, the removal of noise from an observed image is referred to as image denoising. This paper focuses on image denoising for analyzing observed image sequences.

In the literature, there has been extensive discussion on image denoising (Qiu [4]). Many early methods in the computer science literature are based on the Markov random field (MRF) framework, in which observed image intensities of an image are assumed to have the Markov property that the observed intensity at a given pixel depends only on the observed intensities in a neighborhood of the given pixel (Geman and Geman [5]). Then, if the true image is assumed to have a prior distribution which is also an MRF, its posterior distribution would be an MRF too, and consequently, the true image can be estimated by the maximum a posteriori (MAP) estimator (e.g., Geman and Geman [5], Besag [6], Fessler et al. [7]). Other popular image denoising methods include those based on diffusion equations (e.g., Perona and Malik [8], Weickert [9]), total variation (Beck and Teboulle [10], Rudin et al. [11], Yuan et al. [12]), wavelet transformations (e.g., Chang et al. [13], Mrázek [14]), jump regression analysis (e.g., Gijbels et al. [15], Qiu [16], Qiu [17], Qiu and Mukherjee [18]), adaptive weights smoothing (e.g., Polzehl and Spokoiny [19]), spatial adaption (e.g., Kervrann and Boulanger [20]) and more. Besides noise removal, edge-preserving is important for image denoising because edges are important structures of the images. Some of the methods mentioned above can preserve edges well, such as the ones based on jump regression analysis, total variation, and wavelet transformations. Thorough surveys of popular edge-preserving image denoising methods can be found in Jain and Tyagi [21] and Qiu [4].

Although there are already some existing methods for edge-preserveing image denoising, almost all of them handle observed images taken at a single time point. So far, we have not found much discussion about denoising image sequences, which is the focus of the current paper. A given image sequence often describes a gradual change in appearance over time, subject to the underlying process. For instance, the sequence of images of the Las Vegas area acquired by the Landsat satellite (cf., Figure 1) describes the gradual change of the Earth’s surface in that area over time. As mentioned above, two consecutive images in the sequence acquired by the current Landsat satellite are only about 16 days apart. So, their difference should be very small. However, the images could be substantially different after a long period of time, as shown in Figure 1. In such applications, it should be reasonable to assume that edge locations in different images either do not change or change gradually over time. To handle such image sequences, the neighboring images should be useful when denoising the image at a given time point, or information in neighboring images should be shared during image denoising. By noticing such features of image sequences, we propose an edge-preserving image denoising procedure for analyzing image sequences in this paper. Our proposed method is based on the jump regression analysis (JRA) used for regression modeling when the underlying regression function has jumps or other singularities (Qiu [22]). It is a local smoothing procedure, and the possible spatio-temporal correlation in the observed image data has been accommodated properly in its construction. Both theoretical arguments and numerical studies show that this method works well in various different cases.

The remaining parts of the article are organized as follows. The proposed method is described in detail in Section 2. Its statistical properties and the numerical studies about its performance in different finite-sample cases are presented in Section 3. Several concluding remarks are provided in Section 4. Some technical details are given in Appendix A.

2. Materials and Methods

This section describes our proposed method in two parts. A JRA model for describing an image sequence and the model estimation are discussed in Section 2.1. Selection of several parameters used in model estimation is discussed in Section 2.2.

2.1. JRA Model and Its Estimation

To describe an image sequence, let us consider the following JRA model:

$$Z_{ijk}=f(x_i,y_j;t_k)+{\epsilon }_{ijk},i=1,2,\dots ,n_x,j=1,2,\dots ,n_y,k=1,2,\dots ,n_t,$$

(1)

where $Z_{ijk}$ is the observed image intensity level at the $(i,j)$-th pixel $(x_i,y_j)$ and at the k-th time point $t_k$, $f(x_i,y_j;t_k)$ is the true image intensity level, and ${\epsilon }_{ijk}$ is the pointwise random noise with mean 0 and variance ${\sigma }^2$. In model (1), spatio-temporal data correlation is allowed, namely, $\left\{{\epsilon }_{ijk}\right\}$ could be correlated over $i,j$ and k. For image data, the pixel locations are usually regularly spaced. Without loss of generality, it is assumed that they are equally spaced in the design space $\Omega =[0,1]\times [0,1]$, namely, $(x_i,y_j)=(i/n_x,j/n_y)$, for all i and j, where $n_x$ and $n_y$ are the numbers of rows and columns, respectively. The observation times $\{t_k,k=1,2,\dots ,n_t\}$ are also assumed to be equally spaced in the time interval $[0,1]$. The true image intensity function $f(x,y;t)$, for $(x,y)\in \Omega$, is continuous in the design space $\Omega$ at each $t\in [0,1]$, except on the edges where it has jumps.

To estimate the unknown image intensity function $f(x,y;t)$ in model (1), we consider using a local smoothing method, instead of a global smoothing method (e.g., smoothing spline method), because of a large amount of data involved in the current problem. Likewise, it has been well-discussed in the JRA literature that conventional smoothing methods (e.g., conventional local kernel smoothing methods) would not work well for estimating models like (1) where the true image intensity function $f(x,y;t)$ has jumps at the edges, because the jumps would be blurred by such conventional methods (cf., Qiu [22]). In this paper, we suggest a jump-preserving local smoothing method for estimating (1), described in detail below. For a given point $(x,y;t)\in \Omega \times [0,1]$, define a local neighborhood

$$\begin{array}{cc}\hfill O(x,y;t)=& \{\left(x^{{}^{\prime }},y^{{}^{\prime }};t^{{}^{\prime }}\right):\left(x^{{}^{\prime }},y^{{}^{\prime }};t^{{}^{\prime }}\right)\in \Omega \times [0,1],\hfill \\ & \sqrt{\frac{{(x^{{}^{\prime }}-x)}^2}{h_x^2}+\frac{{(y^{{}^{\prime }}-y)}^2}{h_y^2}}\le 1,|t^{{}^{\prime }}-t|/h_t\le 1\},\hfill \end{array}$$

where $h_x$, $h_y$ and $h_t$ are the bandwidths in the $x-$, $y-$, and $t-$axis, respectively. In $O(x,y;t)$, we first consider the following local linear kernel (LLK) smoothing procedure (Fan and Gijbels [23]):

$$\begin{array}{cc}\hfill \underset{a,b,c,d}{min}\sum _{i=1}^{n_x}\sum _{j=1}^{n_y}\sum _{k=1}^{n_t}& {\left\{Z_{ijk}-\left[a+b(x_i-x)+c(y_j-y)+d(t_k-t)\right]\right\}}^2\hfill \\ \hfill & K\left(\frac{x_i-x}{h_x},\frac{y_j-y}{h_y}\right)K\left(\frac{t_k-t}{h_t}\right),\hfill \end{array}$$

(2)

where $K\left(v\right)$ is a density kernel function with the support $\{v:|v|\le 1\}$. The solutions to $(a,b,c,d)$ of the minimization problem (2) are denoted as $\widehat{a}(x,y;t)$, $\widehat{b}(x,y;t)$, $\widehat{c}(x,y;t)$, and $\widehat{d}(x,y;t)$, respectively. It can be checked that they have the following expressions:

$$\left[\begin{array}{c}\widehat{a}(x,y;t)\\ \widehat{b}(x,y;t)\\ \widehat{c}(x,y;t)\\ \widehat{d}(x,y;t)\end{array}\right]={\left[\begin{array}{cccc}{m}_{000}& m_{100}& m_{010}& m_{001}\\ m_{100}& m_{200}& m_{110}& m_{101}\\ m_{010}& m_{110}& m_{020}& m_{011}\\ m_{001}& m_{101}& m_{011}& m_{002}\end{array}\right]}^{-1}\left[\begin{array}{c}\sum _{ijk}{Z}_{ijk}{K}_{ijk}\\ \sum _{ijk}(x_i-x)Z_{ijk}{K}_{ijk}\\ \sum _{ijk}(y_j-y)Z_{ijk}{K}_{ijk}\\ \sum _{ijk}(t_k-t)Z_{ijk}{K}_{ijk}\end{array}\right],$$

(3)

where ${\sum }_{ijk}$ denotes ${\sum }_{i=1}^{n_x}{\sum }_{j=1}^{n_y}{\sum }_{k=1}^{n_t}$, $K_{ijk}$ denotes $K(\frac{x_i-x}{h_x},\frac{y_j-y}{h_y})K\left(\frac{t_k-t}{h_t}\right)$, and $m_{rsl}={\sum }_{ijk}{(x_i-x)}^r{(y_j-y)}^s{(t_k-t)}^l{K}_{ijk}$, for $r,s,l=0,1,2$. The LLK estimator of $f(x,y;t)$ is defined to be $\widehat{a}(x,y;t)$. The estimated gradient direction of $f(x,y;t)$ at $(x,y;t)$ is $\widehat{G}(x,y;t)={(\widehat{b}(x,y;t),\widehat{c}(x,y;t),\widehat{d}(x,y;t))}^{\prime }$ which indicates the direction in which the estimated plane in $O(x,y;t)$ by the LLK procedure (2) increases the fastest. If there is an edge surface in $O(x,y;t)$, then $\widehat{G}(x,y;t)$ would be (approximately) orthogonal to that surface.

In cases when there are no edges in the neighborhood $O(x,y;t)$, $\widehat{a}(x,y;t)$ would be a good estimate of $f(x,y;t)$. Otherwise, it cannot be a good estimate because $\widehat{a}(x,y;t)$ is a weighted average of all observed image intensities in $O(x,y;t)$, the jumps in the image intensity surface would be smoothed out in the weighted average, and the estimate $\widehat{a}(x,y;t)$ would be biased for estimating $f(x,y;t)$. To overcome that limitation, we consider the following one-sided smoothing idea. Let $O(x,y;t)$ be divided into two parts $O^{\left(1\right)}(x,y;t)$ and $O^{\left(2\right)}(x,y;t)$ by a plane that passes $(x,y;t)$ and is perpendicular to $\widehat{G}(x,y;t)$. See Figure 2 for an example.

Then, in cases when there is an edge surface in $O(x,y;t)$, that plane would be (approximately) parallel to the edge surface. Consequently, at least one of $O^{\left(1\right)}(x,y;t)$ and $O^{\left(2\right)}(x,y;t)$ would be (mostly) located on a single side of the edge surface in such cases. Now, let us consider the following one-sided LLK smoothing procedure: for $l=1,2$,

$$\begin{array}{cc}\hfill \underset{a,b,c,d}{min}\sum _{(x_i,y_j;t_k)\in O^{\left(l\right)}(x,y;t)}& {\left\{Z_{ijk}-\left[a+b(x_i-x)+c(y_j-y)+d(t_k-t)\right]\right\}}^2\hfill \\ \hfill & K\left(\frac{x_i-x}{h_x},\frac{y_j-y}{h_y}\right)K\left(\frac{t_k-t}{h_t}\right).\hfill \end{array}$$

(4)

The solutions of (4) to $(a,b,c,d)$ are denoted as $({\widehat{a}}^{\left(l\right)}(x,y;t),$${\widehat{b}}^{\left(l\right)}(x,y;t),$${\widehat{c}}^{\left(l\right)}(x,y;t),$${\widehat{d}}^{\left(l\right)}(x,y;t))$, for $l=1,2$. Intuitively, when there are no edges in $O(x,y;t)$, $\widehat{a}(x,y;t)$, ${\widehat{a}}^{\left(1\right)}(x,y;t)$ and ${\widehat{a}}^{\left(2\right)}(x,y;t)$ are all consistent estimates of $f(x,y;t)$ under some regular conditions. In such cases, $\widehat{a}(x,y;t)$ would be preferred since it averages more observations and consequently it would have a smaller variance. When there are edges in $O(x,y;t)$, $\widehat{a}(x,y;t)$ would not be a good estimate of $f(x,y;t)$ as explained above, but one of ${\widehat{a}}^{\left(1\right)}(x,y;t)$ and ${\widehat{a}}^{\left(2\right)}(x,y;t)$ should estimate $f(x,y;t)$ well. Therefore, in all cases, at least one of the three estimators $\widehat{a}(x,y;t)$, ${\widehat{a}}^{\left(1\right)}(x,y;t)$ and ${\widehat{a}}^{\left(2\right)}(x,y;t)$ should estimate $f(x,y;t)$ well.

Next, we need to choose a good estimator from $\widehat{a}(x,y;t)$, ${\widehat{a}}^{\left(1\right)}(x,y;t)$ and ${\widehat{a}}^{\left(2\right)}(x,y;t)$ based on the observed data, which is not straightforward, partly because we do not know in advance whether there are edges in the neighborhood $O(x,y;t)$ and whether the edges are mostly contained in $O^{\left(1\right)}(x,y;t)$ or $O^{\left(2\right)}(x,y;t)$ if the answer to the first question is positive. To overcome this difficulty, let us consider the following weighted residual mean squares (WRMS) of the fitted local plane by the LLK procedure (2):

$$\begin{array}{cc}\hfill e(x,y;t)=& \{\sum _{ijk}[Z_{ijk}-\widehat{a}(x,y;t)-\widehat{b}(x,y;t)(x_i-x)-\widehat{c}(x,y;t)(y_j-y)-\hfill \\ \hfill & \widehat{d}(x,y;t)(t_k-t){]}^2{K}_{ijk}\}/\sum _{ijk}{K}_{ijk}.\hfill \end{array}$$

(5)

The above WRMS measures how well the fitted local plane describes the observed data in $O(x,y;t)$. If there are edges in $O(x,y;t)$, this quantity would be relatively large, due mainly to the jumps in the image intensity surface. Otherwise, it would be relatively small. So, the quantity $e(x,y;t)$ contains useful information about the existence of edges in $O(x,y;t)$. Similarly, we can define WRMS values for the two one-sided local planes fitted in $O^{\left(1\right)}(x,y;t)$ and $O^{\left(2\right)}(x,y;t)$. They are denoted as $e^{\left(1\right)}(x,y;t)$ and $e^{\left(2\right)}(x,y;t)$. Based on these WRMS values, we define our edge-preserving estimator of $f(x,y;t)$ to be

$$\begin{array}{cc}\hfill \widehat{f}(x,y;t)& =\widehat{a}(x,y;t)I(D(x,y;t)\le u)\hfill \\ \hfill & +{\widehat{a}}^{\left(1\right)}(x,y;t)I(D(x,y;t)>u)I(e^{\left(1\right)}(x,y;t)<e^{\left(2\right)}(x,y;t))\hfill \\ \hfill & +{\widehat{a}}^{\left(2\right)}(x,y;t)I(D(x,y;t)>u)I(e^{\left(1\right)}(x,y;t)>e^{\left(2\right)}(x,y;t))\hfill \\ \hfill & +\frac{{\widehat{a}}^{\left(1\right)}(x,y;t)+{\widehat{a}}^{\left(2\right)}(x,y;t)}{2}I(D(x,y;t)>u)I(e^{\left(1\right)}(x,y;t)=e^{\left(2\right)}(x,y;t)),\hfill \end{array}$$

(6)

where $D(x,y;t)=max(e(x,y;t)-e^{\left(1\right)}(x,y;t),e(x,y;t)-e^{\left(2\right)}(x,y;t))$, $I(·)$ is the indicator function, and $u>0$ is a threshold parameter. By (6), it is obvious that $\widehat{f}(x,y;t)$ is defined to be one of $\widehat{a}(x,y;t)$, ${\widehat{a}}^{\left(1\right)}(x,y;t)$ and ${\widehat{a}}^{\left(2\right)}(x,y;t)$. The quantity $\widehat{a}(x,y;t)$, which is obtained from the entire neighborhood $O(x,y;t)$, is chosen if the observed data indicate no edges in $O(x,y;t)$, supported by the event $D(x,y;t)\le u$. Otherwise, one of the two one-sided quantities, ${\widehat{a}}^{\left(1\right)}(x,y;t)$ and ${\widehat{a}}^{\left(2\right)}(x,y;t)$, with a smaller WRMS value is chosen. Although, theoretically, the event $(e^{\left(1\right)}(x,y;t)=e^{\left(2\right)}(x,y;t))$ would have probability zero of happening, the last term on the right-hand-side of (6) is still included for completeness of the definition of $\widehat{f}(x,y;t)$ and for the consideration that $e^{\left(1\right)}(x,y;t)$ and $e^{\left(2\right)}(x,y;t)$ could be considered the same in certain algorithms when their values are close.

2.2. Parameter Selection

In our proposed method described in Section 2.1, there are four parameters; $h_x$, $h_y$, $h_t$ and u, that need to be chosen properly in advance. For that purpose, it is natural to consider the cross validation (CV) procedure, especially in the current research problem where the observed data are quite large in size. However, it has been well-demonstrated in the literature that the conventional CV procedure would not work well in cases when the observed data are autocorrelated, because it cannot effectively distinguish the data correlation structure from the mean structure (cf., Altman [24], Opsomer et al. [25]). In the current problem, spatio-temperal data correlation is possible in almost all applications. Thus, the conventional CV procedure is not feasible in such cases. In the univariate regression setup, Brabanter et al. [26] suggested a modified CV procedure for choosing smoothing parameters in cases with correlated data. This procedure is generalized here for choosing the parameters $h_x$, $h_y$, $h_t$ and u used in the proposed method, which is described below. Let the modified CV score for choosing $h_x$, $h_y$, $h_t$ and u be defined as

$$CV(h_x,h_y,h_t,u)=\frac{1}{n_x{n}_y{n}_t}\sum _{ijk}{\left[{\widehat{f}}_{-\left(ijk\right)}(x_i,y_j;t_k)-Z(x_i,y_j;t_k)\right]}^2,$$

(7)

where ${\widehat{f}}_{-\left(ijk\right)}(x_i,y_j;t_k)$ is the leave-one-out estimate of $f(x_i,y_j;t_k)$ by (2)–(6) after the observation $Z_{ijk}$ is removed from the estimation process and after the kernel function is replaced by the so-called $ϵ$-optimal bimodal kernel function $K_{ϵ}\left(v\right)$ defined to be

$$K_{ϵ}\left(v\right)=\frac{4}{4-3ϵ-{ϵ}^3}\times \left\{\begin{array}{cc}\frac{3}{4}(1-v^2)I\left(\right|v|\le 1),\hfill & if\left|v\right|\ge ϵ,\hfill \\ \frac{3(1-{ϵ}^2)}{4ϵ}\left|v\right|,\hfill & if\left|v\right|<ϵ,\hfill \end{array}\right.$$

(8)

where $0<ϵ<1$ is a parameter. Based on a large simulation study, Brabanter et al. [26] suggested choosing $ϵ$ to be 0.1, which is adopted in this paper. Then, by the above modified CV procedure, (7) and (8), the parameters $h_x$, $h_y$, $h_t$ and u can be chosen by minimizing the modified CV score $CV(h_x,h_y,h_t,u)$.

3. Results

3.1. Statistical Properties

In this part, we discuss some statistical properties of the proposed edge-preserving image sequence denoising method (2)–(6). First, we have the following proposition.

Proposition 1.

Assume that i) the kernel function $K\left(v\right)$ used in (2) is a Lipschitz-1 continuous density function, and ii) the noise terms $\{{\epsilon }_{ijk},i=1,2,\dots ,n_x,j=1,2,\dots ,n_y,k=1,2,\dots ,n_t\}$ in model (1) form a strong mixing stochastic process with the following strong mixing coefficients:

$$\begin{array}{ccc}\hfill \alpha \left(d\right)& =& \underset{\left(ijk\right),\left(i^{{}^{\prime }}{j}^{{}^{\prime }}{k}^{{}^{\prime }}\right)}{sup}\underset{A,B}{sup}\{|P(A\cap B)-P\left(A\right)P\left(B\right)|,A\in \sigma \left({\epsilon }_{ijk}\right),B\in \sigma \left({\epsilon }_{i^{{}^{\prime }}{j}^{{}^{\prime }}{k}^{{}^{\prime }}}\right),\hfill \\ \hfill & & max\left\{\right|i-i^{{}^{\prime }}|,|j-j^{{}^{\prime }}|,|k-k^{{}^{\prime }}\left|\right\}>d\},\hfill \end{array}$$

which have the property that $\alpha \left(d\right)\le c_1{\sigma }^2{\rho }^{c_{2}d}$, where $c_1,c_2>0$ and $0<\rho <1$ are constants, and iii) $E\left({\epsilon }_{111}^6\right)<\infty$. Let $N=n_x{n}_y{n}_t$, $H=h_x{h}_y{h}_t$, $n_{min}=min(n_x,n_y,n_t)$, and $h_{min}=min(h_x,h_y,h_t)$. Then, for any $(x,y;t)\in {\Omega }_h=[h_x,1-h_x]\times [h_y,1-h_y]\times [h_t,1-h_t]$, we have

$$|\frac{1}{NH}\sum _{ijk}K\left(\frac{x_i-x}{h_x},\frac{y_i-y}{h_y}\right)K\left(\frac{t_i-t}{h_t}\right)-1|=O\left(\frac{1}{n_{min}{h}_{min}}\right),$$

$$E\left[|\frac{1}{NH}\sum _{ijk}{\epsilon }_{ijk}K\left(\frac{x_i-x}{h_x},\frac{y_i-y}{h_y}\right)K\left(\frac{t_i-t}{h_t}\right){|}^2\right]=O\left(\frac{1}{NH}\right),$$

$$E\left[|\frac{1}{NH}\sum _{ijk}({\epsilon }_{ijk}^2-{\sigma }^2)K\left(\frac{x_i-x}{h_x},\frac{y_i-y}{h_y}\right)K\left(\frac{t_i-t}{h_t}\right){|}^2\right]=O\left(\frac{1}{NH}\right).$$

Based on the results in Proposition 1, we can derive the following properties of the LLK estimates defined in (3).

Theorem 1.

Besides the conditions in Proposition 1, we further assume that the true image intensity function $f(x,y;t)$ has continuous first-order partial derivatives with respect to x, y and t in the design space Ω except at the edge curves. Then, for any $(x,y;t)\in {\Omega }_h\setminus J_h$, we have

$$\left[\begin{array}{c}\widehat{a}(x,y;t)\\ \widehat{b}(x,y;t)\\ \widehat{c}(x,y;t)\\ \widehat{d}(x,y;t)\end{array}\right]=\left[\begin{array}{c}f(x,y;t)\\ f_x^{{}^{\prime }}(x,y;t)\\ f_y^{{}^{\prime }}(x,y;t)\\ f_t^{{}^{\prime }}(x,y;t)\end{array}\right]+\left[\begin{array}{c}O(h_x^2+h_y^2+h_t^2)\\ O\left(\frac{h_x^2+h_y^2+h_t^2}{h_x}\right)\\ O\left(\frac{h_x^2+h_y^2+h_t^2}{h_y}\right)\\ O\left(\frac{h_x^2+h_y^2+h_t^2}{h_t}\right)\end{array}\right]+\left[\begin{array}{c}{O}_p\left(\frac{1}{\sqrt{NH}}\right)\\ O_p\left(\frac{1}{h_x\sqrt{NH}}\right)\\ O_p\left(\frac{1}{h_y\sqrt{NH}}\right)\\ O_p\left(\frac{1}{h_t\sqrt{NH}}\right)\end{array}\right].$$

for any $(x,y,t)\in J_h\setminus S_h$, we have

$$\begin{array}{cc}\hfill \left[\begin{array}{c}\widehat{a}(x,y;t)\\ \widehat{b}(x,y;t)\\ \widehat{c}(x,y;t)\\ \widehat{d}(x,y;t)\end{array}\right]=& \left[\begin{array}{c}{f}_{-}(x_{\tau },y_{\tau };t_{\tau })+d_{\tau }{\xi }_{000}^{\left(2\right)}\\ \frac{d_{\tau }}{{\xi }_{200}{h}_x}{\xi }_{100}^{\left(2\right)}\\ \frac{d_{\tau }}{{\xi }_{020}{h}_y}{\xi }_{010}^{\left(2\right)}\\ \frac{d_{\tau }}{{\xi }_{002}{h}_t}{\xi }_{001}^{\left(2\right)}\end{array}\right]+\left[\begin{array}{c}O\left(\sqrt{h_x^2+h_y^2+h_t^2}\right)\\ O\left(\frac{\sqrt{h_x^2+h_y^2+h_t^2}}{h_x}\right)\\ O\left(\frac{\sqrt{h_x^2+h_y^2+h_t^2}}{h_y}\right)\\ O\left(\frac{\sqrt{h_x^2+h_y^2+h_t^2}}{h_t}\right)\end{array}\right]+\left[\begin{array}{c}{O}_p\left(\frac{1}{\sqrt{NH}}\right)\\ O_p\left(\frac{1}{h_x\sqrt{NH}}\right)\\ O_p\left(\frac{1}{h_y\sqrt{NH}}\right)\\ O_p\left(\frac{1}{h_t\sqrt{NH}}\right)\end{array}\right],\hfill \end{array}$$

(9)

where ${\xi }_{rsl}={\int }_{\Omega \times [0,1]}{u}^r{v}^s{w}^{l}K(u,v)K\left(w\right)dudvdw$, ${\xi }_{rsl}^{\left(2\right)}={\int }_{Q^{\left(2\right)}}{u}^r{v}^s{w}^{l}K(u,v)K\left(w\right)dudvdw$, for $r,s,l=0,1,2$, J is the closure of the set of all jump points of $f(x,y;t)$, $J_h=\{(x,y;t):(x,y;t)\in {\Omega }_h,\sqrt{{(x-x^{*})}^2/h_x^2+{(y-y^{*})}^2/h_y^2}\le 1,|t-t^{*}|/h_t\le 1,forany(x^{*},y^{*},t^{*})\in J\}$, S is the set of singular points in J, including the crossing points of two or more edges, points on an edge surface at which the edge surface does not have a unique tangent surface, and points in J at which the jump sizes in $f(x,y;t)$ are zero, $S_h=\{(x,y;t):(x,y;t)\in {\Omega }_h$, $\sqrt{{(x-x^{*})}^2/h_x^2+{(y-y^{*})}^2/h_y^2}\le 1,|t-t^{*}|/h_t\le 1,forany(x^{*},y^{*},t^{*})\in S\}$, $(x_{\tau },y_{\tau };t_{\tau })\in J\setminus S$ is the projection of $(x,y;t)$ to J with the Euclidean distance between the two points being $c\sqrt{h_x^2+h_y^2+h_t^2}$, for a constant $0<c<1$, and $f_{-}(x_{\tau },y_{\tau };t_{\tau })$ is the smaller one of the two one-sided limits of $f(x,y;t)$ at $(x_{\tau },y_{\tau };t_{\tau })$. In cases when $O(x,y;t)$ contains jumps, without loss of generality, it is assumed that $O(x,y;t)$ is divided by the edge surface into two parts $I_1$ and $I_2$ with a positive jump size $d_{\tau }$ from $I_1$ to $I_2$ at $(x_{\tau },y_{\tau };t_{\tau })$, and $Q^{\left(1\right)}$ and $Q^{\left(2\right)}$ are the two corresponding parts in the support of $K(u,v)K\left(w\right)$.

The next two theorems establish the consistency of the proposed edge-preserving image denoising procedure (2)–(6). First, we have the following theorem about the WRMS values defined in (5).

Theorem 2.

Assume that the conditions in Theorem 1 are satisfied, $h_x^2+h_y^2+h_t^2=o\left(1\right)$, $(h_x^2+h_y^2+h_t^2)/h_{min}=o\left(1\right)$, $1/\left(NH\right)=o\left(1\right)$ and $1/\left(NH{h}_{min}^2\right)=o\left(1\right)$. Then, we have the following results: for any $(x,y;t)\in {\Omega }_h\setminus J_h$,

$$\begin{array}{cc}\hfill e(x,y;t)& ={\sigma }^2+o_p\left(1\right),\hfill \\ \hfill e^{\left(l\right)}(x,y;t)& ={\sigma }^2+o_p\left(1\right),forl=1,2;\hfill \end{array}$$

(10)

for any $(x,y;t)\in J_h\setminus S_h$,

$$\begin{array}{cc}\hfill e(x,y;t)& ={\sigma }^2+d_{\tau }{C}_{\tau }^2+o_p\left(1\right),\hfill \\ \hfill e^{\left(l\right)}(x,y;t)& ={\sigma }^2+d_{\tau }{\left[C_{\tau }^{\left(l\right)}\right]}^2+o_p\left(1\right),forl=1,2,\hfill \end{array}$$

(11)

where

$$\begin{array}{ccc}\hfill C_{\tau }& =& (\int \int {\int }_{Q^{\left(1\right)}}{\left[{\xi }_{000}^{\left(2\right)}+\frac{{\xi }_{100}^{\left(2\right)}}{{\xi }_{200}}u+\frac{{\xi }_{010}^{\left(2\right)}}{{\xi }_{020}}v+\frac{{\xi }_{001}^{\left(2\right)}}{{\xi }_{002}}w\right]}^{2}K(u,v)K\left(w\right)dudvdw+\hfill \\ \hfill & & \int \int {\int }_{Q^{\left(2\right)}}{\left[1-{\xi }_{000}^{\left(2\right)}-\frac{{\xi }_{100}^{\left(2\right)}}{{\xi }_{200}}u-\frac{{\xi }_{010}^{\left(2\right)}}{{\xi }_{020}}v-\frac{{\xi }_{001}^{\left(2\right)}}{{\xi }_{002}}w\right]}^{2}K(u,v)K\left(w\right)dudvdw{)}^{1/2}.\hfill \end{array}$$

and

$$\begin{array}{ccc}\hfill C_{\tau }^{\left(l\right)}& =& (2\int \int {\int }_{Q^{\left(1l\right)}}{\left[B_{0l}+\frac{B_{1l}}{{\xi }_{200}}u+\frac{B_{2l}}{{\xi }_{020}}v+\frac{B_{3l}}{{\xi }_{002}}w\right]}^{2}K(u,v)K\left(w\right)dudvdw+\hfill \\ & & 2\int \int {\int }_{Q^{\left(2l\right)}}{\left[1-B_{0l}-\frac{B_{1l}}{{\xi }_{200}}u-\frac{B_{2l}}{{\xi }_{020}}v-\frac{B_{3l}}{{\xi }_{002}}w\right]}^{2}K(u,v)K\left(w\right)dudvdw{)}^{1/2}.\hfill \\ \hfill \end{array}$$

with the quantities $Q^{\left(1l\right)}$, $Q^{\left(2l\right)}$, $B_{0l}$, $B_{1l}$, $B_{2l}$ and $B_{3l}$ defined as follows. Let $\overrightarrow{g}=(\frac{d_{\tau }}{{\xi }_{200}{h}_x}{\xi }_{100}^{\left(2\right)}$, $\frac{d_{\tau }}{{\xi }_{020}{h}_y}{\xi }_{010}^{\left(2\right)}$, $\frac{d_{\tau }}{{\xi }_{002}{h}_t}{\xi }_{001}^{\left(2\right)})$. Then, from (9), $\overrightarrow{g}$ is actually the asymptotic direction of the gradient vector $\widehat{G}(x,y;t)$. Let ${\tilde{O}}^{\left(l\right)}(x,y;t)$, for $l=1,2$, be two halves of the neighborhood $O(x,y;t)$ separated by a plane passing the point $(x,y;t)$ in the direction perpendicular to $\overrightarrow{g}$ and ${\tilde{Q}}^{\left(l\right)}$ be the two corresponding parts in the support of $K(u,v)K\left(w\right)$. Then, $Q^{\left(1l\right)}=Q^{\left(1\right)}\cap {\tilde{Q}}^{\left(l\right)}$, $Q^{\left(2l\right)}=Q^{\left(2\right)}\cap {\tilde{Q}}^{\left(l\right)}$, $B_{0l}=\int \int {\int }_{Q^{\left(2l\right)}}K(u,v)K\left(w\right)dudvdw$, $B_{1l}=\int \int {\int }_{Q^{\left(2l\right)}}uK(u,v)K\left(w\right)dudvdw$, $B_{2l}=\int \int {\int }_{Q^{\left(2l\right)}}vK(u,v)K\left(w\right)dudvdw$, and $B_{3l}=\int \int {\int }_{Q^{\left(2l\right)}}wK(u,v)K\left(w\right)dudvdw$, for $l=1,2$.

Theorem 3.

Under the conditions in Theorem 2 and the extra assumption that threshold parameter $u=u_N\to 0$ as $N\to \infty$, we have, for any $(x,y;t)\in {\Omega }_h$,

$$\widehat{f}(x,y;t)=f(x,y;t)+o_p\left(1\right).$$

The proofs of these theoretical results are given in Appendix A.

3.2. Numerical Studies

In this part, we study the numerical performance of our proposed method for denoising an image sequence. First, we consider a simulation example in which the true image intensity function in model (1) has the following expression:

$$f(x,y;t)=\left\{\begin{array}{cc}-2{(x-0.5)}^2-2{(y-0.5)}^2-0.1sin\left(2\pi t\right)+1,\hfill & ifr(x,y;t)\le 0.{25}^2,\hfill \\ -2{(x-0.5)}^2-2{(y-0.5)}^2-0.1sin\left(2\pi t\right),\hfill & otherwise,\hfill \end{array}\right.$$

where $r(x,y;t)={(x-0.5)}^2+{(y-0.5)}^2+0.01sin\left(2\pi t\right)$, $(x,y)\in \Omega =[0,1]\times [0,1]$, and $t\in [0,1]$. At a given value of t, $f(x,y;t)$ has a circular edge curve $r(x,y;t)=0.{25}^2$ with a constant jump size 1 in $f(x,y;t)$ at the edges. The radius of the circular edge curve, $\sqrt{0.{25}^2-0.01sin\left(2\pi t\right)}$, changes periodically over $t\in [0,1]$. The image intensity function $f(x,y;t)$ at $t=0.01$ and 0.25 and its temporal profile $f(0.25,0.25;t)$ are shown in Figure 3. It can be seen that both the image intensity level at a given pixel and the edge curve change gradually when t changes in $[0,1]$.

In model (1), the random errors $\{{\epsilon }_{ijk},i=1,2,\dots ,n_x,j=1,2,\dots ,n_y,k=1,2,\dots ,n_t\}$ are generated by the function spatialnoise() in the R-package neuRosim (cf., Welvaert et al. [27]). In that R function, there are two parameters $\rho$ and $\sigma$ to specify in advance, where $\rho$ controls the data autocorrelation in all three dimensions and $\sigma$ is the common standard deviation of the random errors. In all our examples, $\sigma$ is fixed at 0.1, 0.2 or 0.3, and $\rho$ is fixed at 0.1, 0.3 or 0.5, to study the possible impact of data noise level and data correlation on the performance of the proposed method. Without loss of generality, we set $n_x=n_y$ in all examples. In the model estimation procedure (2)–(6), we set $h_x=h_y$, and the kernel function $K\left(v\right)$ is chosen to be the following truncated Gaussian density function:

$$K\left(v\right)=\left\{\begin{array}{cc}\frac{exp(-v^2/2)-exp(-0.5)}{2\pi -3\pi exp(-0.5)},\hfill & if\left|v\right|\le 1,\hfill \\ 0,\hfill & otherwise.\hfill \end{array}\right.$$

In cases when $\sigma =0.1$, 0.2 or 0.3, $n_x=64$ or 128, $n_t=50$ or 100, $\rho =0.1$, 0.3 or 0.5, the MSE values of the estimator $\widehat{f}(x,y;t)$ defined in (6) are presented in

Table 1. In each entry, MSE of $\widehat{f}(x,y;t)$ in (6) is presented in the first line with its standard error (in parenthesis); the corresponding values of $(h_x,h_t,u)$ chosen by the modified CV procedure (7) and (8) is presented in the second line; the optimal MSE is presented in the third line with its standard error (in parenthesis); the optimal values of $(h_{xy},h_t,u)$ are presented in the fourth line. MSE in the table has been multiplied by ${10}^3$ and standard error has been multiplied by ${10}^5$.

		${\mathit{n}}_{\mathit{t}}=50$		${\mathit{n}}_{\mathit{t}}=100$
$\mathbf{\sigma }$	$\mathbf{\rho }$	${\mathit{n}}_{\mathit{x}}=\mathbf{64}$	${\mathit{n}}_{\mathit{x}}=\mathbf{128}$	${\mathit{n}}_{\mathit{x}}=\mathbf{64}$	${\mathit{n}}_{\mathit{x}}=\mathbf{128}$
0.1	0.1	$0.65\left(0.80\right)$	$0.30\left(0.25\right)$	$0.48\left(0.43\right)$	$0.26\left(0.10\right)$
		(0.03, 0.10, 0.05)	(0.03, 0.08, 0.025)	(0.03, 0.10, 0.05)	(0.02, 0.07, 0.05)
		$0.32\left(0.46\right)$	$0.20\left(0.14\right)$	$0.37\left(0.36\right)$	$0.19\left(0.08\right)$
		(0.04, 0.07, 0.025)	(0.03, 0.05, 0.025)	(0.03, 0.08, 0.025)	(0.02, 0.05, 0.025)
	0.3	$0.60\left(0.45\right)$	$0.33\left(0.16\right)$	$0.59\left(0.39\right)$	$0.33\left(0.15\right)$
		(0.04, 0.10, 0.05)	(0.03, 0.07, 0.025)	(0.03, 0.10, 0.05)	(0.02, 0.07, 0.025)
		$0.49\left(0.35\right)$	$0.30\left(0.16\right)$	$0.50\left(0.37\right)$	$0.29\left(0.22\right)$
		(0.04, 0.08, 0.025)	(0.03, 0.06, 0.025)	(0.03, 0.08, 0.025)	(0.03, 0.04, 0.025)
	0.5	$1.25\left(1.24\right)$	$0.80\left(0.22\right)$	$0.81\left(0.55\right)$	$0.64\left(0.21\right)$
		(0.03, 0.10, 0.05)	(0.02, 0.07, 0.025)	(0.03, 0.10, 0.05)	(0.02, 0.04, 0.025)
		$0.77\left(0.65\right)$	$0.49\left(0.24\right)$	$0.74\left(0.46\right)$	$0.45\left(0.25\right)$
		(0.04, 0.09, 0.025)	(0.03, 0.06, 0.025)	(0.03, 0.09, 0.025)	(0.03, 0.04, 0.025)
0.2	0.1	$1.14\left(1.13\right)$	$0.68\left(0.38\right)$	$1.02\left(0.74\right)$	$0.56\left(0.26\right)$
		(0.04, 0.10, 0.025)	(0.03, 0.08, 0.025)	(0.04, 0.10, 0.025)	(0.03, 0.07, 0.025)
		$1.11\left(0.86\right)$	$0.66\left(0.33\right)$	$0.93\left(0.71\right)$	$0.54\left(0.31\right)$
		(0.04, 0.09, 0.025)	(0.03, 0.07, 0.025)	(0.04, 0.08, 0.025)	(0.03, 0.05, 0.025)
	0.3	$1.69\left(0.91\right)$	$1.03\left(0.54\right)$	$1.32\left(1.08\right)$	$0.78\left(0.41\right)$
		(0.04, 0.10, 0.025)	(0.03, 0.08, 0.025)	(0.04, 0.10, 0.025)	(0.03, 0.07, 0.025)
		$1.69\left(1.24\right)$	$1.03\left(0.54\right)$	$1.29\left(1.12\right)$	$0.78\left(0.41\right)$
		(0.04, 0.11, 0.025)	(0.03, 0.08, 0.025)	(0.04, 0.09, 0.025)	(0.03, 0.07, 0.025)
	0.5	$3.25\left(1.74\right)$	$2.88\left(0.78\right)$	$1.95\left(1.85\right)$	$2.61\left(0.58\right)$
		(0.04, 0.07, 0.025)	(0.02, 0.07, 0.025)	(0.04, 0.09, 0.025)	(0.02, 0.04, 0.025)
		$2.59\left(2.23\right)$	$1.54\left(1.32\right)$	$1.91\left(1.78\right)$	$1.21\left(0.43\right)$
		(0.05, 0.10, 0.025)	(0.04, 0.09, 0.025)	(0.04, 0.11, 0.025)	(0.03, 0.08, 0.025)
0.3	0.1	$2.32\left(1.91\right)$	$1.26\left(1.03\right)$	$1.59\left(0.81\right)$	$0.92\left(0.34\right)$
		(0.05, 0.13, 0.025)	(0.04, 0.09, 0.025)	(0.04, 0.11, 0.025)	(0.03, 0.08, 0.025)
		$2.28\left(2.58\right)$	$1.26\left(1.03\right)$	$1.59\left(0.65\right)$	$0.92\left(0.34\right)$
		(0.05, 0.11, 0.025)	(0.04, 0.09, 0.025)	(0.04, 0.10, 0.025)	(0.03, 0.08, 0.025)
	0.3	$3.15\left(2.28\right)$	$1.72\left(1.37\right)$	$2.26\left(1.53\right)$	$1.36\left(0.50\right)$
		(0.05, 0.13, 0.025)	(0.04, 0.09, 0.025)	(0.04, 0.11, 0.025)	(0.03, 0.08, 0.025)
		$3.14\left(2.45\right)$	$1.71\left(1.52\right)$	$2.21\left(1.31\right)$	$1.33\left(0.41\right)$
		(0.05, 0.14, 0.025)	(0.04, 0.10, 0.025)	(0.04, 0.13, 0.025)	(0.04, 0.09, 0.025)
	0.5	$6.78\left(3.46\right)$	$6.81\left(2.00\right)$	$4.18\left(2.72\right)$	$6.33\left(1.43\right)$
		(0.04, 0.09, 0.05)	(0.02, 0.07, 0.05)	(0.04, 0.10, 0.025)	(0.02, 0.04, 0.05)
		$4.46\left(4.94\right)$	$2.48\left(2.38\right)$	$3.18\left(3.42\right)$	$1.88\left(0.56\right)$
		(0.06, 0.16, 0.025)	(0.05, 0.11, 0.025)	(0.05, 0.14, 0.025)	(0.04, 0.10, 0.025)

, along with the corresponding parameters $h_x$, $h_t$ and u selected by the modified CV procedure (7) and (8). In each case considered, the MSE value is computed based on 10 replicated simulations. For comparison purposes, the optimal MSE value of the estimator $\widehat{f}(x,y;t)$, when its parameters ($h_x$, $h_t$ and u) are chosen such that the MSE value reaches the minimum in each case considered, is also presented in the table, along with the corresponding parameter values. From the table, we can draw the following conclusions. (i) The MSE values are smaller when either $n_x$ or $n_t$ is larger, which confirms the consistency results discussed in Section 3.1. (ii) When $\rho$ is larger (i.e., the spatio-temporal data correlation is stronger), the MSE values are larger. So, data correlation does have an impact on the performance of the proposed method, which is intuitively reasonable. (iii) By comparing the MSE and the optimal MSE values, we can see that the MSE values are usually larger than their optimal values, but their differences are not that big in almost all cases considered. This conclusion indicates that the modified CV procedure (7) and (8) for determining the values of the parameters $(h_x,h_t,u)$ is quite effective. (iv) The parameter values chosen by the modified CV procedure (7) and (8) are quite close to the optimal parameter values in most cases considered.

Next, we compare our proposed method, denoted as NEW, with some alternative methods described below. The first alternative method is the conventional LLK procedure (2), by which $f(x,y;t)$ is estimated by $\widehat{a}(x,y;t)$ defined in (3). Its bandwidths are chosen by the conventional CV procedure, without considering any possible spatio-temporal data correlation. As explained in Section 2.1, this estimator would blur edges while removing noise. The second alternative method is to use $\widehat{a}(x,y;t)$ for estimating $f(x,y;t)$, but its bandwidths are chosen by the modified CV procedure (7) and (8). The above two alternative methods are denoted as LLK-C and LLK, respectively, where LLK-C denotes the first conventional LLK procedure that does not accommodate data correlation. The third alternative method is the one by Gijbels et al. [15] which is used for edge-preserving image denoising of a single image. To apply this method to the current problem, individual images collected at different time points can be denoised by it separately. This method assumes that the observed image intensities at different pixels are independent of each other, and thus their bandwidths can be chosen by the conventional CV procedure. This method is denoted as GLQ. The fourth alternative method is to use $\widehat{f}(x,y;t)$ in (6) to estimate $f(x,y;t)$, but the parameters $(h_x,h_t,u)$ are chosen by the conventional CV procedure. This method is denoted as NEW-C. By considering all these four alternative methods (i.e., LLK-C, LLK, GLQ and NEW-C), we can check whether the current problem to denoise an image sequence can be handled properly by the conventional LLK procedure with or without using the modified CV procedure, by an existing edge-preserving image denoising method designed for denoising a single image, or by the proposed method without considering the possible spatio-temporal data correlation. To evaluate their performance, in addition to the regular MSE criterion, we also consider the following edge-preservation (EP) criterion originally discussed in Hall and Qiu [28]:

$$EP\left(\widehat{f}\right)=|JS\left(\widehat{f}\right)-JS\left(f\right)|/JS\left(f\right),$$

where

$$\begin{array}{cc}\hfill JS\left(f\right)=& \frac{1}{(n_x-2)(n_y-2)(n_t-2)}\sum _{i=2}^{n_x-1}\sum _{j=2}^{n_y-1}\sum _{k=2}^{n_t-1}({[f(x_{i+1},y_j;t_k)-f(x_{i-1},y_j;t_k)]}^2+\hfill \\ \hfill & {[f(x_i,y_{j+1};t_k)-f(x_i,y_{j-1};t_k)]}^2+{[f(x_i,y_j;t_{k+1})-f(x_i,y_j;t_{k-1})]}^2{)}^{1/2},\hfill \end{array}$$

and JS($\widehat{f}$) is defined similarly. According to Hall and Qiu [28], JS(f) is a reasonable measure of the cumulative jump magnitude of f at the edge locations. So, $EP\left(\widehat{f}\right)$ provides a measure of the percentage of the cumulative jump magnitude of f that has been lost during data smoothing by using the estimator $\widehat{f}$. By this explanation, the smaller its value, the better. In cases when $\sigma =0.1$, 0.2 or 0.3, $n_x=128$, $n_t=100$, and $\rho =0.1$, 0.3 or 0.5, the MSE and EP values of the related methods are presented in

Table 2. In each entry, the first line is the MSE value with its standard error (in parenthesis), and the second line is the EP value. MSE values in the table are in the unit of ${10}^3$ and the standard error values are in the unit of ${10}^5$.

$\mathit{\sigma }$	$\mathit{\rho }$	LLK-C	LLK	GLQ	NEW-C	NEW
0.1	0.1	$2.06\left(0.08\right)$	$2.10\left(0.06\right)$	$0.60\left(0.18\right)$	$0.24\left(0.11\right)$	$0.26\left(0.10\right)$
		$73.68\%$	$18.43\%$	$28.24\%$	$12.32\%$	$7.48\%$
	0.3	$3.04\left(0.14\right)$	$2.28\left(0.09\right)$	$0.95\left(0.18\right)$	$2.93\left(0.40\right)$	$0.33\left(0.15\right)$
		$124.48\%$	$34.40\%$	$43.69\%$	$131.28\%$	$10.58\%$
	0.5	$3.89\left(0.24\right)$	$3.23\left(0.21\right)$	$1.42\left(0.42\right)$	$3.77\left(0.48\right)$	$0.64\left(0.21\right)$
		$141.47\%$	$95.86\%$	$57.40\%$	$148.17\%$	$28.86\%$
0.2	0.1	$4.16\left(0.25\right)$	$2.93\left(0.15\right)$	$1.51\left(0.38\right)$	$0.86\left(0.25\right)$	$0.56\left(0.26\right)$
		$142.65\%$	$51.78\%$	$54.40\%$	$39.01\%$	$9.14\%$
	0.3	$9.39\left(0.52\right)$	$3.67\left(0.25\right)$	$2.87\left(0.51\right)$	$9.60\left(0.78\right)$	$0.78\left(0.41\right)$
		$291.31\%$	$82.84\%$	$94.59\%$	$295.72\%$	$15.08\%$
	0.5	$12.80\left(0.94\right)$	$11.21\left(0.86\right)$	$7.75\left(1.32\right)$	$13.12\left(1.16\right)$	$2.61\left(0.58\right)$
		$326.38\%$	$289.71\%$	$203.86\%$	$334.62\%$	$84.24\%$
0.3	0.1	$7.88\left(0.57\right)$	$3.94\left(0.26\right)$	$3.17\left(0.86\right)$	$1.01\left(0.37\right)$	$0.92\left(0.34\right)$
		$235.43\%$	$82.24\%$	$73.18\%$	$23.36\%$	$15.41\%$
	0.3	$19.97\left(1.15\right)$	$5.56\left(0.50\right)$	$12.36\left(0.63\right)$	$19.97\left(1.16\right)$	$1.36\left(0.50\right)$
		$461.12\%$	$133.33\%$	$261.31\%$	$461.13\%$	$25.78\%$
	0.5	$27.64\left(2.09\right)$	$23.75\left(1.92\right)$	$15.75\left(1.71\right)$	$28.04\left(2.29\right)$	$6.33\left(1.43\right)$
		$514.22\%$	$458.82\%$	$292.50\%$	$518.16\%$	$144.58\%$

. From the table, it can be seen that the proposed method NEW has the smallest MSE values with quite large margins among all five methods in all cases considered, except the case when $\sigma =0.1$ and $\rho =0.1$ where NEW-C has a lightly smaller MSE value than that of NEW due to the weak data correlation in that case. Likewise, NEW has much smaller EP values in all cases considered, compared to the four competing methods. This example confirms that it is necessary to consider edge-preserving procedures when denoising image sequences and the possible spatio-temporal data correlation should be taken into account during the denoising process. It also confirms the benefit to share useful information among neighboring images when denoising an image sequence.

In the cases when $\sigma =0.2$ and $\rho =0.1$, 0.3 or 0.5, Figure 4 shows the observed images at $t=0.5$ in the first column, and the denoised images by the methods LLK-C, LLK, GLQ, NEW-C and NEW in columns 2–6. From the figure, it can be seen that the denoised images by NEW are the best in removing noise and preserving edges. As a comparison, the denoised images by LLK-C, and NEW-C are quite noisy because their selected bandwidths by the conventional CV procedure are relatively small due to the fact the conventional CV procedure cannot distinguish the data correlation from the mean structure, as discussed in Section 2.2. The denoised images by LLK are quite blurry because the method does not take the edges into account when denoising the images. The denoised images by GLQ are quite blurry as well since GLQ denoises individual images at different time points separately and the serial data correlation is ignored in this method.

Next, we apply the proposed method NEW and the four alternative methods LLK-C, LLK, GLQ and NEW-C to a sequence of cell images that records the vasculogenesis process. The sequence has 100 images, and each image has $128\times 128$ pixels. A detailed description of the data can be found in Svoboda et al. [29]. The 1st, 50th and 100th images of the sequence are shown in Figure 5.

In the image denoising literature, to test the noise removal ability of a image denoising method, it is a common practice to add random noise at a certain level to the test images and then apply the image denoising method to the noisy test images (cf., Gijbels et al. [15]). To follow this convention, spatio-temporally correlated noise is first generated using the R-package neuRosim and then added to the sequence of 100 cell images described above. When generating the noise, $\sigma$ is chosen to be 0.1, 0.2 or 0.3 and $\rho$ is chosen to be 0.1, 0.3 or 0.5, as in the simulation examples presented above. The MSE and EP values of the five image denoising methods based on 10 replicated simulations are presented in

Table 3. Results for denoising a sequence of 100 cell images. In each entry, the first line is the MSE value and its standard error (in parenthesis), and the second line is the EP value. MSE values in the table are in the unit of ${10}^3$ and the standard errors are in the unit of ${10}^5$.

$\mathit{\sigma }$	$\mathit{\rho }$	LLK-C	LLK	GLQ	NEW-C	NEW
0.1	0.1	$1.69\left(0.11\right)$	$0.97\left(0.08\right)$	$1.67\left(0.12\right)$	$1.69\left(0.12\right)$	$1.35\left(0.12\right)$
		$63.30\%$	$5.53\%$	$18.88\%$	$63.31\%$	$18.52\%$
	0.3	$2.36\left(0.16\right)$	$1.43\left(0.14\right)$	$1.94\left(0.18\right)$	$2.36\left(0.16\right)$	$1.51\left(0.19\right)$
		$77.54\%$	$31.64\%$	$25.72\%$	$77.55\%$	$7.28\%$
	0.5	$3.21\left(0.25\right)$	$2.82\left(0.24\right)$	$2.28\left(0.29\right)$	$3.21\left(0.25\right)$	$1.92\left(0.31\right)$
		$88.68\%$	$75.95\%$	$30.68\%$	$88.68\%$	$10.11\%$
0.2	0.1	$3.22\left(17.00\right)$	$1.47\left(5.54\right)$	$3.93\left(0.29\right)$	$3.22\left(17.00\right)$	$1.67\left(0.25\right)$
		$85.64\%$	$13.57\%$	$76.53\%$	$85.64\%$	$16.28\%$
	0.3	$8.71\left(0.56\right)$	$2.34\left(0.35\right)$	$5.00\left(0.43\right)$	$8.71\left(0.56\right)$	$2.17\left(0.45\right)$
		$189.74\%$	$42.07\%$	$91.44\%$	$189.75\%$	$4.88\%$
	0.5	$12.12\left(0.94\right)$	$10.35\left(0.88\right)$	$6.41\left(0.86\right)$	$12.14\left(0.96\right)$	$4.48\left(0.90\right)$
		$213.90\%$	$187.93\%$	$102.68\%$	$214.07\%$	$59.86\%$
0.3	0.1	$3.16\left(0.50\right)$	$2.01\left(0.28\right)$	$5.47\left(0.53\right)$	$3.16\left(0.50\right)$	$1.93\left(0.40\right)$
		$47.15\%$	$22.46\%$	$54.20\%$	$47.15\%$	$10.91\%$
	0.3	$19.30\left(1.23\right)$	$4.29\left(0.71\right)$	$10.11\left(0.85\right)$	$19.30\left(1.23\right)$	$2.82\left(0.77\right)$
		$308.32\%$	$79.75\%$	$161.91\%$	$308.32\%$	$14.37\%$
	0.5	$26.96\left(2.09\right)$	$22.88\left(1.95\right)$	$13.36\left(1.82\right)$	$27.00\left(2.13\right)$	$8.75\left(1.85\right)$
		$345.91\%$	$306.28\%$	$180.35\%$	$346.14\%$	$113.48\%$

. From the table, it can be seen that NEW still has smaller MSE and EP values in this example, compared to the four competing methods, except in a small number of cases when $\sigma$ and $\rho$ are relatively small.

The 50th observed test image after the spatio-temporally correlated noise with $\rho =0.1$, 0.3 or 0.5 being added is shown in the first column of Figure 6. The denoised images by the five methods LLK-C, LLK, GLQ, NEW-C and NEW are shown in columns 2–6 of the figure. It can be seen that similar conclusions to those from Figure 4 can be made here, and the denoised images by NEW look reasonably well, as the algorithm work well in removing noise and preserving edges.

Finally, we apply the five methods considered in the above examples to a sequence of Landsat images of the Salton Sea region. The Salton Sea is the largest inland lake located at the southern border of California, US, and has a great impact on the local ecosystem (Shuford et al. [30]). The Landsat images used here were taken during the time period of 27 May 2000 and 24 December 2001. There are a total of 20 images collected at roughly equally-spaced time points, and each image has $100\times 100$ pixels. In this example, we consider the case when $\sigma =0.3$ and $\rho =0.3$. The MSE values of the five methods LLK-C, LLK, GLQ, NEW-C, and NEW calculated in the same way as before are $9.70$, $4.78$, $12.03$, $9.77$, and $4.82$, respectively. Their EP values are respectively $85.54\%$, $20.18\%$, $109.91\%$, $86.15\%$, and $19.14\%$. So, we can see that NEW method has the best edge-preserving performance among the five methods in this example, and NEW and LLK have the best overall noise removal performance. The 10th noisy observed test image taken on 28 April 2001 and its denoised versions by the five methods are shown in Figure 7. It can be seen from the figure that the denoised images by the methods LLK-C, GLQ, and NEW-C are still quite noisy, and the noise in the images generated by NEW and LLK is mostly removed while the edges are preserved reasonably well.

4. Conclusions

In this paper, we have described our proposed edge-preserving image denoising method for handling image sequences. Some major features of the proposed method include (i) helpful information in neighboring images is shared during image denoising, (ii) edge structures in the observed images can be preserved when removing noise, and (iii) possible sptio-temporal data correlation can be accommodated in the related local smoothing procedure. Theoretical arguments given in Section 3.1 and numerical studies presented in Section 3.2 show that the proposed method works well in various cases considered. There are still some issues about the proposed method for future research. For instance, in the proposed local smoothing procedure (2)–(6), each of the bandwidths $(h_x,h_y,h_t)$ is chosen by the modified CV procedure (7) and (8) to be the same in the entire design space $\Omega \times [0,1]$. Intuitively, relatively small bandwidths are preferred at places where the image intensity surface $f(x,y;t)$ has large curvature and relatively large bandwidths are preferred at places where the curvature of $f(x,y;t)$ is small. Thus, in some applications where the curvature of $f(x,y;t)$ could change quite dramatically in the design space, variable bandwidths might be helpful. Such issues will be studied carefully in our future research.