Interpolation Filter Design Based on All-Phase DST and Its Application to Image Demosaicking

Abstract

Based on a deep understanding of all-phase digital filter (APDF) design and all-phase biorthogonal transform (APBT), this paper will further study the windowed all-phase digital filter (WAPDF) and windowed all-phase biorthogonal transform (WAPBT), discuss the principle of the WAPBT, and provide a unified construction method of the all-phase transform (APT). Based on a type of orthogonal transform, i.e., discrete sine transform (DST), an interpolation filter called an all-phase DST (APDST) filter is constructed and used for image demosaicking; it is compared with bilinear interpolation and all-phase inverse discrete cosine transform (APIDCT) interpolation filters, to test its performance in image interpolation and provide analysis and discussion. The experimental results show that APIDCT and APDST filters with a size of 7 × 7 are similar in interpolation performance, but better than the bilinear interpolation method. In addition to its use in image interpolation demosaicking, the low-pass filter designed in this paper can also be widely used in image interpolation, image denoising, image resizing, and other fields of image processing.

1. Introduction

In the field of image processing, image interpolation and compression are areas that are consistently improving [1]. The conventional mathematical transforms currently used in image compression are the Karhunen–Loève transform (KLT), discrete Fourier transform (DFT), Walsh-Hadamard transform (WHT), discrete cosine transform (DCT), discrete sine transform (DST), discrete wavelet transform (DWT), lapped biorthogonal transform (LBT), etc. Among them, DCT is the core transform of the widely used international standards of image and video compression. In the H.265/HEVC standard [2], along with DCT, DST was first implemented in the transform coding part. HEVC uses DCT-II and 4 × 4 DST-VII. It also uses 4 × 4, 8 × 8, 16 × 16, and 32 × 32 transform unit (TU) sizes. Zhou et al. [3] provided a more detailed overview of the mathematical transforms used in image and video coding. In recent years, to provide a significant improvement in compression performance over HEVC, the joint video experts team (JVET) has launched a project to develop a future video coding (FVC) standard H.266 termed versatile video coding (VVC). In addition to DCT-II and 4 × 4 DST-VII, which are adopted in HEVC [2], an adaptive multiple transform (AMT) scheme is used for residual coding for both inter-coded and intra-coded blocks. VVC uses five DCT/DST-based transform types known as DCT-II, DCT-V, DCT-VIII, DST-I, and DST-VII; and employs 4 × 4, 8 × 8, 16 × 16, 32 × 32, and 64 × 64 TU sizes [4]. The JVET established the first draft of the VVC standard specification and the VVC test model 1 (VTM1) encoding method at its 10th meeting (10–20 April 2018, San Diego, CA, USA). Tseng and Lee [5] developed a color digital image sharpening method by using DST-I and matrix low-pass Butterworth filters.

To solve the problem of the serious blocking artifacts at low bit rates in image compression, Hou et al. [6] deduced the expression of the arbitrary order all-phase biorthogonal transform (APBT) matrix based on the design of the all-phase digital filter (APDF) [7], WHT, DCT, inverse DCT (IDCT), and other orthogonal transforms, and proposed a kind of JPEG-like coding algorithm based on the APBT. Because the APBT has good energy concentration characteristic and high-frequency attenuation characteristic, by using the APBT instead of the DCT image transform with a simple uniform quantization to replace the complex quantization table in JPEG, which was designed in accordance with the visual characteristics of human eyes, the effects of low-frequency fine-grained and high-frequency coarse-grained components can be achieved simultaneously, and the algorithm can be simplified. Experimental results show that when the APBT is used for image coding, the quality of the reconstructed image is better than that of the conventional DCT. Especially at low bit rates, the blocking artifacts can be effectively removed from reconstructed images.

To improve the quality of the reconstructed image, starting from the construction of the windowed all-phase digital filter (WAPDF), Fu et al. [8] constructed an efficient signal transform method, windowed all-phase biorthogonal transform (WAPBT); used the optimization algorithm to design the optimized window function of the WAPBT; obtained a WAPBT matrix that is more in line with the image coding requirements; and proposed JPEG-like image coding framework based on WAPBT. It improved the coding performance. Meanwhile, the algorithm is simpler, due to the uniform quantization.

To improve the speed of APBT-based image coding, Wang et al. [9] used CUDA development tools to implement GPU parallelism of APBT and applied it in the JPEG coding framework, achieving parallel processing of transform, quantization, Huffman coding, inverse quantization, inverse transform, and Huffman decoding in image compression, which increases the image coding speed. Experimental results show that parallel processing of parts with higher computational complexity in image compression can facilitate a hundred times faster acceleration than serial processing based on a CPU. In terms of image quality, parallel and serial algorithms have similar reconstructed images at the same compression rate.

Combining with the DST and APDF, Shan et al. [10] proposed the all-phase discrete sine biorthogonal transform (APDSBT). Moreover, they further improved the APBT theory in order to use it in JPEG coding instead of the conventional DCT, which yields JPEG-like coding based on parallel APDSBT. The parallel APDSBT-JPEG image coding algorithm improves the efficiency of the algorithm while solving the blocking artifacts caused by conventional JPEG coding at low bit rates, which improves the subjective and objective performance compared with that of the conventional JPEG coding.

Wang [11] constructed an all-phase IDCT (APIDCT) filter and applied it to a Bayer-patterned image compression system based on wavelet transform and all-phase interpolation. Based on APBT, Xie et al. [12] proposed a new Bayer-patterned image compression algorithm in which the compression process uses JPEG-like compression, and in the interpolation stage, a new APIDCT interpolation filter is adopted and compared with the conventional interpolation method. Simulation results show that both JPEG-like compression and APIDCT interpolation can further improve Bayer-patterned image compression quality.

Wang et al. [13] compared the image compression performance of different interpolation methods based on structure separation and APBT-JPEG. Compared with the conventional DCT-JPEG, APBT-JPEG uses uniform quantization, eliminating the need for quantization table storage, which reduces complex multiplication. Compared with the performance of the commonly used interpolation methods, i.e., nearest interpolation, bilinear interpolation, and cubic convolution, the performance of the APIDCT interpolation is similar to that of the conventional interpolation method and particularly superior to that of the conventional method at high bit rates. Based on the APIDCBT-JPEG compression algorithm, the composite peak signal-to-noise ratio (CPSNR) between the image reconstructed by different interpolation methods and the original image is compared. Experimental results show that the proposed algorithm exceeds the Bayer-patterned color image compression algorithm based on structure separation and is more suitable for compressing Bayer-patterned images.

Wang et al. [14] proposed an improved linear interpolation method for demosaicking of Bayer-patterned color filter array (CFA) images. Experimental results show that the interpolation performance is more effective than bilinear interpolation. Kim and Jeong [15] proposed a four-direction residual interpolation, obtaining a difference between tentative and original images, in which the edge distortion is decreased to some degree. Shi et al. [16] divided the input image into two parts, a smooth region and an edge region, applying bilinear interpolation to the smooth area, and processing the edge region with multi-directional weighted information for interpolation. Wang and Jeon [17] introduced a gradient-based interpolation algorithm. First, an eight-direction weighted interpolation is applied for G component interpolation, and then the obtained G plane is used for R and B components interpolation. Because more weighting information is obtained, the value of CPSNR is larger.

Based on a deep understanding of APDF design and APBT, this paper will further study the WAPDF and WAPBT, discuss the principle of the WAPBT, and then provide a unified construction method of the all-phase transform (APT). Based on the DST, an interpolation filter named all-phase DST (APDST) filter is constructed and used for image interpolation demosaicking, and it is compared with bilinear interpolation and APIDCT interpolation filters to test its performance for image interpolation and provide analysis and discussion. In this paper, the proposed filter corrects the interpolation coefficient using a bilinear method without considering correlation among color components. Thus, the interpolation performance is better than that of the bilinear method, especially for the G component.

The remainder of this paper is organized as follows. Section 2 gives a unified construction method of the APT based on a deep understanding of APDF and APBT. Based on the windowed all-phase discrete sine transform (WAPDST), Section 3 designs several low-pass filters for image interpolation. Section 4 uses the low-pass filter designed in this paper for color interpolation demosaicking, simulates the test image, provides the experimental results and compares them with those of bilinear interpolation and APIDCT interpolation, and presents analysis and discussion. The conclusion and future work are finally given in Section 5.

2. Unified Construction Method of the All-phase Transform

2.1. APDF

De-blocking or compressing the image with conventional filters or transforms will always result in blocking artifacts at the edges of the block [18]. Although this can be improved by increasing the size of the filter or transform matrix, the amount of computation increases and the blocking artifacts still exist. The reason is that the size of a two-dimensional discrete transform is always finite; moreover, a number of finite non-overlapping blocks are processed separately and then spliced back into a single image. Two-dimensional overlapping filtering overlaps conventional methods and performs filtering after moving left and right or up and down on the original basis, the final output of which is the superposition of these overlapping filters. Since the high-frequency errors of the blocks are reversed after moving in the opposite direction, the blocking artifacts after the overlapping process are greatly reduced.

Design of APDF is the method that uses the maximum overlapping filtering and takes all the segmentation of an input sample into account. As shown in Figure 1, assuming that the length of a segment time series is $N$, then for any one of the input points $x(n)$, there are only $N$ data segments ${\mathit{x}}_n,{\mathit{x}}_{n+1},\cdots ,{\mathit{x}}_{n+N-1}$ that contain the point with different interception phases. The position of $x(n)$ in the $k$th ($k=0,1,\cdots ,N-1$) data segment is $k$, and the $k$th output obtained is $y^{(k)}(n)$ at the time. $x(n)$ traverses all data segments to produce $N$ outputs, and the arithmetic average of $N$ outputs is used as the output $y(n)={\displaystyle \sum _{k=0}^{N-1}{y}^{(k)}(n)}$ according to the input $x(n)$. This is the basic idea of APDF. As data are the continuous input, a continuous output is obtained, eliminating the truncation effect of conventional filtering due to the segmentation. Another outstanding advantage is facilitating the control of the signal frequency. When a certain condition is met, a complete linear phase can be achieved, and the amplitude-frequency characteristic shows the disappearance of passband and stopband ripple and the increase in attenuation. APDF is essentially an $N$-order filter, maximizing $(N-1)/N$, i.e., the overlap of each input, and the corresponding output is obtained after traversing all-phases in the unit circle (angular frequency is $\omega =2k\mathsf{\pi }/N$). A more vivid illustration of “all-phases” is shown in Figure 2.

2.2. APBT

By analyzing the design of APDF in different orthogonal transform domains, such as DFT, DCT, IDCT, WHT, and DST, we can find a commonality: APDF is a zero-phase digital filter that overcomes the Gibbs effect with overlapping digital filtering. If the range of elements of the sequency response vector (or matrix) is extended to the complex field, then the one-dimensional zero-phase digital filter of length $2N-1$ (or the two-dimensional zero-phase digital filter of size $(2N-1)\times (2N-1)$) can find the corresponding sequency response vector by using the transition matrix. Using the transition matrix, the unit impulse response and the sequency response vector (or matrix) of the APDF in the time domain can be connected, as shown in Figure 3.

Simultaneously, because the DFT spectrum directly reflects the frequency domain characteristics of the filter, when designing the APDF, the filter can be designed directly from the corresponding sequency response in the DFT domain according to the frequency response requirements. In other orthogonal transform domains (DCT, IDCT, WHT, DST, etc.), the sequency response cannot directly reflect the frequency response of the filter, so the sequency response cannot be obtained directly from the frequency response requirements. However, if the transition matrix $\mathit{G}$ and the inverse transition matrix ${\mathit{G}}^{-1}$ are used as bridges, the conversion between the sequency response vectors (or matrices) of different orthogonal transform domains can be realized, and the filter design of each orthogonal transform domain can be unified.

If $\mathit{G}$ is used to represent different transition matrices uniformly, ${\mathit{F}}_N$ is the N-dimensional sequency response vector, and $\mathit{h}$ is the unit impulse response of the APDF, then the one-dimensional APDF design formula of different orthogonal transform domains can be uniformly expressed as:

$${\left[{\mathit{h}}_{1/2}\right]}_N={[h(0),h(1),\cdots ,h(N-1)]}^{\mathrm{T}}=\mathit{G}{\mathit{F}}_N.$$

(1)

The two-dimensional APDF design formula can also be uniformly expressed as: ${\mathit{h}}_{1/4}=\mathit{G}{\mathit{F}}_{N\times N}{\mathit{G}}_{}^{\mathrm{T}}$. Among them, ${\mathit{F}}_{N\times N}$ is the sequency response matrix with size of $N\times N$, and ${\mathit{h}}_{1/4}$ is one-fourth of the two-dimensional APDF.

If we think it about the converse, assuming that the unit impulse response of the zero-phase digital filter is known, the sequency response vectors (or matrices) of the different orthogonal transform domains may be found, which helps analyze the characteristics of the sequency response. Since one-dimensional zero-phase filters have symmetry: $\mathit{h}(\mathbf{\tau })=\mathit{h}(-\mathbf{\tau })$, among them $\mathbf{\tau }=0,1,\cdots ,N-1$, the N-dimensional sequency response vector ${\mathit{F}}_N$ of different orthogonal transform domains can be obtained by using ${\mathit{h}}_{1/2}={[h(0),h(1),\cdots ,h(N-1)]}^{\mathrm{T}}$:

$${\mathit{F}}_N={\mathit{G}}^{-1}{\left[{\mathit{h}}_{1/2}\right]}_N.$$

(2)

For two-dimensional zero-phase digital filter $\mathit{h}$, also based on the symmetry, ${\mathit{h}}_{1/4}$ can be used to obtain different orthogonal transform domain sequency response matrix ${\mathit{F}}_{N\times N}={\mathit{G}}^{-1}{\mathit{h}}_{1/4}{\mathit{G}}^{-\mathrm{T}}$. For example, the sequency response vector ${[{\mathit{F}}_N]}_{\mathrm{DFT}}$ of the DFT domain may be converted into a sequency response vector ${[{\mathit{F}}_N]}_{\mathrm{DCT}}={\mathit{G}}_{\mathrm{DCT}}^{-1}{\mathit{G}}_{\mathrm{DFT}}{[{\mathit{F}}_N]}_{\mathrm{DFT}}$ of the DCT domain; the sequency response matrix ${[{\mathit{F}}_{N\times N}]}_{\mathrm{DFT}}$ of the DFT domain may be transformed into a sequency response matrix ${[{\mathit{F}}_{N\times N}]}_{\mathrm{DCT}}={\mathit{G}}_{\mathrm{DCT}}^{-1}({\mathit{G}}_{\mathrm{DFT}}^{}{[{\mathit{F}}_{N\times N}]}_{\mathrm{DFT}}{\mathit{G}}_{\mathrm{DFT}}^{\mathrm{T}}){\mathit{G}}_{\mathrm{DCT}}^{-\mathrm{T}}$ of the DCT domain. Similarly, the N-dimensional sequency response vector (or matrix) of the DFT domain can also be converted into the sequency response vectors (or matrices) of the IDCT, WHT, DST domains.

In conclusion, in the APDF design, the sequency response ${\mathit{F}}_N$ of the orthogonal transform domain and the unit impulse response $\mathit{h}$ of the time domain are connected by a transition matrix $\mathit{G}$. In [6], this transform matrix derived from APDF was called the APBT matrix.

2.3. WAPDF and WAPBT

APDF has the characteristics of the conventional frequency sampling method and the original signal weighting and superposition, along with features of the window function design. Therefore, it has advantages of window adding and frequency domain design. To improve the filter performance with the method of windowing, a slightly wider transition band of a flatter passband and stopband characteristics is used. In the design of the WAPDF in [8], the WAPDF sequency response is related to a unit impulse response in its time domain by a conversion matrix, and this matrix is similar to the DCT matrix, which has a good sequency property and can transform images from the spatial domain into the sequency domain. Thus, this matrix, which is called the WAPBT matrix, can be used in image compression, and the corresponding transform is called WAPBT. When the front and back windows are rectangular, WAPDF is APDF, and the WAPBT matrix is the APBT matrix. Therefore, WAPBT can be seen as a special case of APBT.

When ${\mathbf{\alpha }}_N$ is the WHT matrix, the corresponding matrix $\mathit{W}$ is called the windowed all-phase Walsh-Hadamard transform (WAPWHT); when ${\mathbf{\alpha }}_N$ is the DCT matrix, the corresponding matrix $\mathit{W}$ is called the windowed all-phase discrete cosine transform (WAPDCT); when ${\mathbf{\alpha }}_N$ is the IDCT matrix, the corresponding matrix $\mathit{W}$ is called the windowed all-phase inverse discrete cosine transform (WAPIDCT); and when ${\mathbf{\alpha }}_N$ is the DST matrix, the corresponding matrix $\mathit{W}$ is called WAPDST. In addition, the corresponding WAPBT can also be constructed based on other orthogonal transforms.

In [8], the general form of the WAPBT matrix was deduced and used for grayscale image coding. Starting from the design of APDF, the general form of the WAPBT matrix is deduced under arbitrary orthogonal transforms (DFT, DCT, IDCT, WHT, DST, etc.):

$$\mathit{W}(j,k)=\frac{1}{N}{\displaystyle \sum _{i=j}^{N-1}\left[{\mathit{B}}_N^2(i){\mathit{B}}_N^1(i-j){\mathbf{\alpha }}_N(k,i){\mathbf{\alpha }}_N(k,i-j)\right]},j=0,1,\cdots ,N-1;k=0,1,\cdots ,N-1,$$

(3)

where ${\mathit{B}}_N^1$ and ${\mathit{B}}_N^2$ are the front and back windows of the windowed all-phase digital filter, respectively; $N$ is the order of the filter; and ${\mathbf{\alpha }}_N$ is the corresponding orthogonal transform matrix. ${\mathbf{\alpha }}_N$ is a complex matrix in the DFT domain and a real matrix in other domains.

Next, three important properties of WAPBT will be demonstrated.

Property 1.

When the WAPBT front and back windows meet the condition${\mathit{B}}_N^2(i)\cdot {\mathit{B}}_N^1(i)=L$ ($L$ is a real constant number, $i=0,1,\cdots ,N-1$), elements in the first row are equivalent to constants in the WAPBT matrix, which ensures that after the image transform, the first coefficient in the upper left corner of the WAPBT transform domain is the DC coefficient.

Proof. 

The first row of WAPBT matrix: $\mathit{W}(0,k)=\frac{1}{N}{\displaystyle \sum _{i=0}^{N-1}[{\mathit{B}}_N^2(i)}{\mathit{B}}_N^1(i)]{[{\mathbf{\alpha }}_N^{}(k,i)]}^2$, $k=0,1,\cdots ,N-1$. If the front and back windows meet the condition ${\mathit{B}}_N^2(i){\mathit{B}}_N^1(i)=L$ ($L$ is the real constant, $i=0,1,\cdots ,N-1$), then $\mathit{W}(0,k)=\frac{L}{N}{\displaystyle \sum _{i=0}^{N-1}{[{\mathbf{\alpha }}_N^{}(k,i)]}^2}$, $k=0,1,\cdots ,N-1$. If ${\mathbf{\alpha }}_N$ is the WHT matrix, then $\begin{array}{cc}\mathit{W}(0,k)=L,& k=0,1,\cdots ,N-1\end{array}$; If ${\mathbf{\alpha }}_N$ is the DCT, IDCT, or DST matrix, then $\begin{array}{cc}\mathit{W}(0,k)=\frac{L}{N},& k=0,1,\cdots ,N-1\end{array}$. The remaining elements of the WAPBT matrix are:

$$\begin{array}{cc}\mathit{W}(j,k)=\frac{L}{N}{\displaystyle \sum _{i=j}^{N-1}\left[\frac{{\mathit{B}}_N^1(i-j)}{{\mathit{B}}_N^1(i)}{\mathbf{\alpha }}_N(k,i){\mathbf{\alpha }}_N(k,i-j)\right]}=\frac{L}{N}{\displaystyle \sum _{i=j}^{N-1}\left[\frac{{\mathit{B}}_N^2(i)}{{\mathit{B}}_N^2(i-j)}{\mathbf{\alpha }}_N(k,i){\mathbf{\alpha }}_N(k,i-j)\right]},& \begin{array}{l}j=1,2,\cdots ,N-1,\\ k=0,1,\cdots ,N-1.\end{array}\end{array}$$

(4)

It can be seen from Property 1 that different window sequences will have a great impact on the property of the WAPBT matrix.  ☐

Property 2.

When the WAPBT front and back windows meet the condition${\mathit{B}}_N^2(i)=L\cdot {\mathit{B}}_N^1(i)$, $i=0,1,\cdots ,N-1$, $L$is a constant, and${\mathbf{\alpha }}_N$is an orthogonal transform (for example, WHT, DCT, IDCT, or DST), WAPDF is a zero-phase digital filter, and WAPBT strictly connects the$\mathbf{\alpha }$domain and time domain$\mathit{h}$.

Proof. 

From [8], $\mathit{H}(i,j)={\mathit{B}}_N^2(i)\mathit{A}(i,j){\mathit{B}}_N^1(j)=\frac{1}{N}{\mathit{B}}_N^2(i){\mathit{B}}_N^1(j){\displaystyle \sum _{k=0}^{N-1}{\mathit{F}}_N(k){\mathbf{\alpha }}_N^{}(k,i){\mathbf{\alpha }}_N^{}(k,j)}$, and $\mathit{H}(j,i)=\frac{1}{N}{\mathit{B}}_N^2(j){\mathit{B}}_N^1(i){\displaystyle \sum _{k=0}^{N-1}{\mathit{F}}_N(k){\mathbf{\alpha }}_N^{}(k,j){\mathbf{\alpha }}_N^{}(k,i)}$. Then, we can get: $\frac{\mathit{H}(i,j)}{\mathit{H}(j,i)}=\frac{{\mathit{B}}_N^2(i){\mathit{B}}_N^1(j)}{{\mathit{B}}_N^2(j){\mathit{B}}_N^1(i)}$.

Therefore, if and only if ${\mathit{B}}_N^2(i){\mathit{B}}_N^1(j)={\mathit{B}}_N^2(j){\mathit{B}}_N^1(i)$, for example, if $\frac{{\mathit{B}}_N^2(i)}{{\mathit{B}}_N^1(i)}=\frac{{\mathit{B}}_N^2(j)}{{\mathit{B}}_N^1(j)}$, $\frac{\mathit{H}(i,j)}{\mathit{H}(j,i)}=1$. It can be proved that the necessary and sufficient condition of $\frac{{\mathit{B}}_N^2(i)}{{\mathit{B}}_N^1(i)}=\frac{{\mathit{B}}_N^2(j)}{{\mathit{B}}_N^1(j)}$ is ${\mathit{B}}_N^2(i)=L\cdot {\mathit{B}}_N^1(i)$, where $i=0,1,\cdots ,N-1$ and $L$ is a constant. It can be seen from [8] that under the condition $\mathit{H}(i,j)=\mathit{H}(j,i)$, $\mathit{h}(\mathbf{\tau })=\mathit{h}(-\mathbf{\tau })$, $\mathbf{\tau }=0,1,\cdots ,N-1$. Therefore, WAPDF is a zero-phase digital filter, and the connection between the filter’s unit impulse response $\mathit{h}$ and the N-dimensional sequency response vector ${\mathit{F}}_N$ is completely established by $\mathit{W}$. Thus, under this condition, WAPBT matrix $\mathit{W}$ strictly connects the $\mathbf{\alpha }$ domain and time domain $\mathit{h}$. At this time, elements in the WAPBT matrix are:

$$\begin{array}{cc}\mathit{W}(j,k)=\frac{L}{N}{\displaystyle \sum _{i=j}^{N-1}{\mathit{B}}_N^1(i){\mathit{B}}_N^1(i-j){\mathbf{\alpha }}_N^{}(k,i){\mathbf{\alpha }}_N^{}(k,i-j),}& \begin{array}{l}j=0,1,\cdots ,N-1,\\ k=0,1,\cdots ,N-1.\end{array}\end{array}$$

(5)

In Equations (4) and (5), constant $L$ is multiplied by each element in WAPBT matrix. Therefore, the condition that $L=N$ will not affect the property of the WAPBT matrix.  ☐

Property 3.

When both Property 1 and Property 2 are satisfied and$L=1$, which means that the front and back windows are rectangular windows${\mathit{B}}_N^1(i)={\mathit{B}}_N^2(i)=1$ ($i=0,1,\cdots ,N-1$), the WAPBT transform matrix is the APBT matrix. Correspondingly, in this case, WAPBT is APBT.

Property 3 establishes the connection between WAPBT and APBT. Because APBT can be seen as a special case when a rectangular window is added, the study of the windowed situation has more universal significance.

2.4. APT Derived from APDF Theory

Taking transition matrix $\mathit{G}$ and inverse transition matrix ${\mathit{G}}^{-1}$ as the bridges and combining windowed situation with the non-windowed situation, a windowed filter with good amplitude frequency response can be obtained under the non-windowed situation, which simplifies the filter structure. For example, windowed DCT and windowed IDCT can be converted into the corresponding non-windowed situation:

$${[{\mathit{F}}_N]}_{\mathrm{DCT}}={\mathit{G}}_{\mathrm{DCT}}^{-1}{[{\mathit{h}}_N]}_{\mathrm{WDCT}},{[{\mathit{F}}_N]}_{\mathrm{IDCT}}={\mathit{G}}_{\mathrm{IDCT}}^{-1}{[{\mathit{h}}_N]}_{\mathrm{WIDCT}},$$

(6)

where WDCT denotes the windowed DCT and WIDCT represents the windowed IDCT.

In summary, the sequency response of the transform domain and unit impulse response of the time domain are connected by the transition matrix $\mathit{G}$. Using $\mathit{G}$ as a bridge, the sequency responses of different orthogonal transform domains can also be mutually obtained. In the same way, the windowed case is similar to this. The transition matrix at this time is denoted ${\mathit{G}}_{\mathrm{W}}$. $\mathit{G}$ and ${\mathit{G}}_{\mathrm{W}}$ are uniformly defined as APT matrix $\mathit{A}$.

Definition 1.

For one-dimensional data$\mathit{x}$, we define the APT and the inverse transform as$\mathit{y}=\mathit{A}\mathit{x}$and$\mathit{x}={\mathit{A}}^{-1}\mathit{y}$, respectively. For two-dimensional image matrix$\mathit{X}$, we define the APT and the inverse transform as$\mathit{Y}=\mathit{A}\mathit{X}{\mathit{A}}^{\mathrm{T}}$and$\mathit{X}={\mathit{A}}^{-1}\mathit{Y}{({\mathit{A}}^{-1})}^{\mathrm{T}}$, respectively, where$\mathit{Y}$is the transform coefficient matrix. If transform matrix$\mathit{A}$takes different forms in the orthogonal transform domain, such as DFT, DCT, IDCT, WHT, or DST, the corresponding transforms are defined as APDFT, APDCT, APIDCT, APWHT, and APDST, respectively.

In [6], Hou et al. constructed APBT on the basis of APDF theory. Inspired by this, Fu et al. [8] started from the design of WAPDF and constructed WAPBT by designing the window function to obtain a transform matrix better suited for image coding. APBT and WAPBT are uniformly called APT. In summary, a diagram of APT derived from APDF theory is shown in Figure 4.

3. Interpolation Filter Design Based on APDST

When ${\mathbf{\alpha }}_N$ is the DST-I matrix, the DST matrix $\mathit{S}(i,j)$ with a size of $N\times N$ is shown in Equation (7):

$$\mathit{S}(i,j)=\sqrt{\frac{2}{N+1}}\sin \frac{(i+1)(j+1)\mathsf{\pi }}{N+1},i,j=0,1,\cdots ,N-1.$$

(7)

The DST-I matrix is a symmetric matrix [19], i.e., $\mathit{S}={\mathit{S}}^{\mathrm{T}}$.

When the front and back windows are rectangular windows and ${\mathbf{\alpha }}_N$ is the DST-I matrix, according to Property 1 of WAPBT, substituting Equation (7) into Equation (5) provides the WAPBT as shown in Equation (8):

$$\mathit{W}(j,k)=\{\begin{array}{cc}\frac{1}{N},\hfill & \hfill j=0;k=0,1,2,\cdots ,N-1,\\ \frac{2}{N^2+N}{\displaystyle \sum _{i=j}^{N-1}\sin \frac{(i+1)(k+1)\mathsf{\pi }}{N+1}}\sin \frac{(i-j+1)(k+1)\mathsf{\pi }}{N+1},\hfill & \hfill \begin{array}{c}j=1,2,\cdots ,N-1,\\ k=0,1,\cdots ,N-1.\end{array}\end{array}$$

(8)

When $N=4$,

$${\mathit{W}}_{4\times 4}=\left[\begin{array}{rrrr}\hfill 0.2500& \hfill 0.2500& \hfill 0.2500& \hfill 0.2500\\ \hfill 0.2023& \hfill 0.0773& \hfill -0.0773& \hfill -0.2023\\ \hfill 0.1118& \hfill -0.1118& \hfill -0.1118& \hfill 0.1118\\ \hfill 0.0345& \hfill -0.0905& \hfill 0.0905& \hfill -0.0345\end{array}\right].$$

(9)

When $N=8$,

$${\mathit{W}}_{s\times s}=\left[\begin{array}{cccccccc}0.1250& 0.1250& 0.1250& 0.1250& 0.1250& 0.1250& 0.1250& 0.1250\\ 0.1175& 0.0958& 0.0625& 0.0217& -0.0217& -0.0625& -0.0958& -0.1175\\ 0.0090& 0.0332& -0.0417& -0.0905& -0.0905& -0.0417& 0.0332& 0.0090\\ 0.0747& -0.0273& -0.0833& -0.0438& 0.0438& 0.0833& 0.0273& -0.0747\\ 0.0496& -0.0596& -0.0417& 0.0516& 0.0516& -0.0417& -0.0596& 0.0496\\ 0.0279& -0.0579& 0.0208& 0.0441& -0.0441& -0.0208& 0.0579& -0.0279\\ 0.0122& -0.0352& 0.0417& -0.0187& -0.0187& 0.0417& -0.0352& 0.0122\\ 0.0032& -0.0115& 0.0208& -0.0269& 0.0269& -0.0208& 0.0115& -0.0032\end{array}\right].$$

(10)

The 8 × 8 WAPBT in Equation (10) takes the place of DCT in [8] and is used in the JPEG image coding system. In addition, the simpler uniform quantization is used as the quantization method. A compression coding performance comparable to that of APBT-JPEG has been achieved in the sense of the peak signal-to-noise ratio (PSNR) and mean structural similarity (MSSIM).

There are many methods for designing a two-dimensional diamond interpolation filter, and most of them based on the half-band filter. This paper uses APDF theory based on DST to design a new diamond filter, which is a low-pass interpolation filter.

Let ${\mathit{h}}_{1/2}={[h(0),h(1),\cdots ,h(N-1)]}^{\mathrm{T}}$ be the unit impulse response vector of the one-dimensional APDST filter and ${\mathit{F}}_N$ be a column response vector of length $N$, then the design of one-dimensional APDST filter $\mathit{h}$ of length $2N-1$ consists of Equation (8), Equation (11), and Equation (12):

$${\left[{\mathit{h}}_{1/2}\right]}_N={[h(0),h(1),\cdots ,h(N-1)]}^{\mathrm{T}}=\mathit{W}{\mathit{F}}_N,$$

(11)

$$\mathit{h}(-\mathbf{\tau })=\mathit{h}(\mathbf{\tau }),\mathbf{\tau }=0,1,\cdots ,N-1.$$

(12)

Similarly, the above one-dimensional APDST filter method can be extended to the two-dimensional case. Let ${\mathit{h}}_{(2N-1)\times (2N-1)}$ be a two-dimensional APDST filter unit impulse response matrix and ${\mathit{F}}_{N\times N}$ be the ideal low-pass filter response matrix with an order of $N\times N$, then the design of a two-dimensional APDST filter with an order of $(2N-1)\times (2N-1)$ consists of Equation (13) and Equation (14):

$${\left[{\mathit{h}}_{1/4}\right]}_{N\times N}=\mathit{W}{\mathit{F}}_{N\times N}{\mathit{W}}^{\mathrm{T}},$$

(13)

$$\mathit{h}(m,n)=\mathit{h}(-m,n)=\mathit{h}(m,-n)=\mathit{h}(-m,-n),m,n=0,1,\cdots ,N-1,$$

(14)

where the definition of matrix $\mathit{W}$ is the same as in Equation (8).

In the following section, the design of the APDST filter is taken as an example to illustrate a construction method of a low-pass filter that can be used for image interpolation demosaicking.

When $N=4$, ${\mathit{F}}_{4\times 4}=\left[\begin{array}{cccc}1& 1& 1& 0.5\\ 1& 1& 0.5& 0\\ 1& 0.5& 0& 0\\ 0.5& 0& 0& 0\end{array}\right]$ is substituted with Equation (9) into Equation (13), and the left-bottom quarter of the filter can be obtained, which is:

$${\left[{\mathit{h}}_{1/4}\right]}_{4\times 4}=\left[\begin{array}{cccc}0.5000& 0.1710& 0& 0.0033\\ 0.1710& 0& -0.0313& 0\\ 0& -0.0313& 0& 0.0063\\ 0.0033& 0& 0.0063& 0\end{array}\right].$$

(15)

According to the zero-phase symmetry characteristic of APDF shown in Equation (14), the 7 × 7 APDST filter is obtained:

$${\left[\mathit{h}\right]}_{7\times 7}=\left[\begin{array}{ccccccc}0& 0.0063& 0& 0.0033& 0& 0.0063& 0\\ 0.0063& 0& -0.0313& 0& -0.0313& 0& 0.0063\\ 0& -0.0313& 0& 0.1710& 0& -0.0313& 0\\ 0.0033& 0& 0.1710& 0.5000& 0.1710& 0& 0.0033\\ 0& -0.0313& 0& 0.1710& 0& -0.0313& 0\\ 0.0063& 0& -0.0313& 0& -0.0313& 0& 0.0063\\ 0& 0.0063& 0& 0.0033& 0& 0.0063& 0\end{array}\right].$$

(16)

As an interpolation filter, the two-dimensional APDST filter with an order of 7 × 7 is:

$${\left[{\mathit{h}}^{\prime }\right]}_{7\times 7}=2{\left[\mathit{h}\right]}_{7\times 7}=\left[\begin{array}{ccccccc}0& 0.0125& 0& 0.0066& 0& 0.0125& 0\\ 0.0125& 0& -0.0625& 0& -0.0625& 0& 0.0125\\ 0& -0.0625& 0& 0.3420& 0& -0.0625& 0\\ 0.0066& 0& 0.3420& 1& 0.3420& 0& 0.0066\\ 0& -0.0625& 0& 0.3420& 0& -0.0625& 0\\ 0.0125& 0& -0.0625& 0& -0.0625& 0& 0.0125\\ 0& 0.0125& 0& 0.0066& 0& 0.0125& 0\end{array}\right].$$

(17)

The frequency response perspective is shown in Figure 5. It can be seen from Figure 5 that the interpolation filter is a low-pass filter which is flat in frequency domain and has little leakage at the low-frequency portion. The all-phase interpolation filter has good interpolation performance.

Similarly, an interpolation filter with an arbitrary order of $(2N-1)\times (2N-1)(N\in \mathbb{N})$ can be designed. For example, the filters with sizes of 3 × 3, 5 × 5, and 9 × 9 are shown in Equations (18)–(20):

$${\left[{\mathit{h}}^{\prime }\right]}_{3\times 3}=\left[\begin{array}{ccc}0& 0.25& 0\\ 0.25& 1& 0.25\\ 0& 0.25& 0\end{array}\right],$$

(18)

$${\left[{\mathit{h}}^{\prime }\right]}_{5\times 5}=\left[\begin{array}{ccccc}0& -0.0393& 0& -0.0393& 0\\ -0.0393& 0& 0.3143& 0& -0.0393\\ 0& 0.3143& 1& 0.3143& 0\\ -0.0393& 0& 0.3143& 0& -0.0393\\ 0& -0.0393& 0& -0.0393& 0\end{array}\right],$$

(19)

$${\left[{\mathit{h}}^{\prime }\right]}_{9\times 9}=\left[\begin{array}{ccccccccc}0& -0.0053& 0& -0.0024& 0& -0.0024& 0& -0.0053& 0\\ -0.0053& 0& 0.0232& 0& 0.0124& 0& 0.0232& 0& -0.0053\\ 0& 0.0232& 0& -0.0771& 0& -0.0771& 0& 0.0232& 0\\ -0.0024& 0& -0.0771& 0& 0.3571& 0& -0.0771& 0& -0.0024\\ 0& 0.0124& 0& 0.3571& 1& 0.3571& 0& 0.0124& 0\\ -0.0024& 0& -0.0771& 0& 0.3571& 0& -0.0771& 0& -0.0024\\ 0& 0.0232& 0& -0.0771& 0& -0.0771& 0& 0.0232& 0\\ -0.0053& 0& 0.0232& 0& 0.0124& 0& 0.0232& 0& -0.0053\\ 0& -0.0053& 0& -0.0024& 0& -0.0024& 0& -0.0053& 0\end{array}\right].$$

(20)

As seen in Equations (17)–(20), except for the center of each interpolation template, the sum of each weight coefficient is 1. In addition, each weight coefficient is symmetrical at the center vertical line, the value of which is interlaced to 0. This can achieve interpolation of the G component in Bayer CFA images to reconstruct the G plane. If the filter above is rotated by 45°, then the R and B planes can be reconstructed by image interpolation in the same way. Finally, the full-color RGB images are obtained.

4. Image Demosaicking Using Interpolation Filters Based on APDST

The block diagram of the proposed image demosaicking method is shown in Figure 6. To show the performance of the APDST interpolation filters designed in this paper, 24 Kodak test images [20] were chosen for image interpolation experiments. These test images are shown in Figure 7, and their size is 768 × 512 or 512 × 768. For the boundary of the images, we extend it with a symmetric boundary extension. All experiments were performed in MATLAB R2012a.

For APIDCT and APDST, a filter with a size of 7 × 7 is used to interpolate the G component, while the R and B components are interpolated using the rotated filter with a size of 11 × 11 (the same 7 × 7 filter is rotated by 45°). For bilinear interpolation, the R, G, and B components are interpolated by the bilinear method. The experimental results of the PSNR and CPSNR to the 24 images are given in

Table 1. Comparisons of PSNR and CPSNR among bilinear interpolation [14], APIDCT interpolation [11], and APDST interpolation of Kodak images with a size of 512 × 768 or 768 × 512 in PNG format.

Test Images	Bilinear [14]				APIDCT [11]				APDST
	PSNR/dB			CPSNR/dB	PSNR/dB			CPSNR/dB	PSNR/dB			CPSNR/dB
	R	G	B		R	G	B		R	G	B
Figure 7a	24.97	29.47	25.30	26.16	24.88	31.19	25.11	26.26	24.87	31.23	25.08	26.25
Figure 7b	31.85	35.92	31.43	32.65	31.73	37.12	31.20	32.66	31.73	37.16	31.17	32.65
Figure 7c	32.63	36.61	32.68	33.62	32.64	37.82	32.39	33.67	32.65	37.96	32.36	33.69
Figure 7d	32.20	36.22	32.20	33.17	32.37	37.48	32.19	33.43	32.39	37.55	32.19	33.45
Figure 7e	25.51	29.18	25.99	26.62	25.62	30.65	25.96	26.89	25.62	30.71	25.94	26.89
Figure 7f	26.41	30.88	26.74	27.59	26.24	32.33	26.41	27.58	26.23	32.47	26.38	27.57
Figure 7g	31.99	35.98	32.02	32.97	32.31	37.55	32.17	33.40	32.34	37.71	32.16	33.44
Figure 7h	22.46	27.35	22.72	23.68	22.34	29.54	22.50	23.78	22.33	29.61	22.47	23.76
Figure 7i	31.06	35.48	31.33	32.21	31.20	37.40	31.26	32.49	31.21	37.55	31.26	32.51
Figure 7j	31.09	35.13	31.18	32.11	31.12	36.67	31.03	32.28	31.13	36.80	31.03	32.29
Figure 7k	28.07	32.20	28.40	29.20	27.98	33.34	28.18	29.24	27.97	33.38	28.16	29.23
Figure 7l	31.18	36.20	32.05	32.66	31.22	37.71	31.83	32.78	31.25	38.03	31.82	32.82
Figure 7m	23.02	26.45	23.12	23.93	22.90	26.99	22.91	23.89	22.89	26.99	22.89	23.88
Figure 7n	28.07	31.88	28.21	29.07	28.04	33.14	28.07	29.19	28.03	33.19	28.04	29.18
Figure 7o	29.74	34.42	30.40	31.09	29.74	35.58	30.16	31.15	29.74	35.71	30.14	31.16
Figure 7p	29.93	34.36	29.70	30.87	29.68	35.90	29.39	30.82	29.66	36.01	29.36	30.81
Figure 7q	31.43	34.57	30.76	31.96	31.42	35.40	30.76	32.10	31.41	35.43	30.76	32.10
Figure 7r	27.19	30.45	26.94	27.93	27.17	31.19	26.82	28.00	27.17	31.20	26.80	27.99
Figure 7s	27.02	31.76	27.13	28.15	26.97	34.00	26.99	28.33	26.96	34.06	26.97	28.32
Figure 7t	29.93	33.80	29.75	30.81	29.80	34.98	29.74	30.92	29.82	35.20	29.75	30.96
Figure 7u	27.56	31.50	27.53	28.51	27.48	32.73	27.35	28.58	27.46	32.78	27.33	28.57
Figure 7v	29.80	33.33	29.33	30.49	29.75	34.50	29.23	30.61	29.75	34.57	29.21	30.61
Figure 7w	33.43	37.56	33.62	34.50	33.69	39.30	33.84	34.96	33.73	39.57	33.84	35.01
Figure 7x	26.34	29.43	25.33	26.72	26.23	30.17	25.20	26.73	26.22	30.20	25.18	26.73

. It can be seen from Table 1 that the performance of the APIDCT is close to that of the APDST filter with the size of 7 × 7, but both of them are better than the bilinear interpolation method, especially the PSNR of the G component, which is approximately 1.5 dB higher than that from the bilinear method. The experimental results of the structural similarity (SSIM) and MSSIM to the 24 images are given in

Table 2. Comparisons of SSIM and MSSIM among bilinear interpolation [14], APIDCT interpolation [11], and APDST interpolation of Kodak images with a size of 512 × 768 or 768 × 512 in PNG format.

Test Images	Bilinear [14]				APIDCT [11]				APDST
	SSIM			MSSIM	SSIM			MSSIM	SSIM			MSSIM
	R	G	B		R	G	B		R	G	B
Figure 7a	0.7498	0.9120	0.7626	0.8081	0.7444	0.9316	0.7563	0.8108	0.7440	0.9321	0.7552	0.8104
Figure 7b	0.8884	0.9432	0.8627	0.8981	0.8816	0.9513	0.8579	0.8969	0.8823	0.9516	0.8573	0.8970
Figure 7c	0.9188	0.9654	0.9135	0.9326	0.9147	0.9677	0.9099	0.9308	0.9156	0.9691	0.9100	0.9316
Figure 7d	0.8936	0.9511	0.8901	0.9116	0.8908	0.9563	0.8881	0.9117	0.8921	0.9568	0.8886	0.9125
Figure 7e	0.9460	0.9400	0.8485	0.8782	0.8485	0.9552	0.8484	0.8840	0.8485	0.9558	0.8480	0.8841
Figure 7f	0.8160	0.9280	0.7981	0.8474	0.8076	0.9419	0.7869	0.8455	0.8086	0.9444	0.7865	0.8465
Figure 7g	0.9428	0.9756	0.9417	0.9534	0.9428	0.9773	0.9426	0.9543	0.9440	0.9788	0.9430	0.9553
Figure 7h	0.7789	0.9222	0.7768	0.8259	0.7717	0.9425	0.7674	0.8272	0.7713	0.9433	0.7665	0.8270
Figure 7i	0.9051	0.9584	0.8966	0.9200	0.8991	0.9605	0.8902	0.9166	0.9006	0.9618	0.8917	0.9180
Figure 7j	0.9038	0.9597	0.8948	0.9194	0.8987	0.9643	0.8891	0.9174	0.8996	0.9654	0.8904	0.9185
Figure 7k	0.8410	0.9366	0.8306	0.8694	0.8346	0.9490	0.8231	0.8689	0.8344	0.9500	0.8225	0.8690
Figure 7l	0.8806	0.9568	0.8852	0.9075	0.8727	0.9597	0.8789	0.9038	0.8744	0.9632	0.8794	0.9057
Figure 7m	0.7199	0.8826	0.7159	0.7728	0.7158	0.8964	0.7069	0.7730	0.7156	0.8968	0.7062	0.7728
Figure 7n	0.8424	0.9349	0.8374	0.8716	0.8403	0.9471	0.8355	0.8743	0.8404	0.9478	0.8351	0.8745
Figure 7o	0.8963	0.9543	0.8926	0.9144	0.8913	0.9529	0.8871	0.9104	0.8926	0.9557	0.8878	0.9120
Figure 7p	0.8496	0.9426	0.8402	0.8775	0.8423	0.9540	0.8323	0.8762	0.8425	0.9556	0.8320	0.8767
Figure 7q	0.9100	0.9599	0.9055	0.9251	0.9089	0.9635	0.9051	0.9258	0.9093	0.9636	0.9054	0.9261
Figure 7r	0.8472	0.9315	0.8245	0.8677	0.8479	0.9415	0.8234	0.8709	0.8479	0.9416	0.8232	0.8709
Figure 7s	0.8511	0.9358	0.8298	0.8722	0.8472	0.9454	0.8234	0.8720	0.8479	0.9461	0.8239	0.8727
Figure 7t	0.9218	0.9598	0.8727	0.9181	0.9127	0.9543	0.8646	0.9106	0.9174	0.9606	0.8668	0.9149
Figure 7u	0.8709	0.9436	0.8514	0.8886	0.8673	0.9485	0.8471	0.8877	0.8678	0.9502	0.8475	0.8885
Figure 7v	0.8677	0.9384	0.8481	0.8848	0.8652	0.9467	0.8459	0.8860	0.8660	0.9479	0.8458	0.8866
Figure 7w	0.9473	0.9749	0.9465	0.9562	0.9442	0.9736	0.9463	0.9547	0.9455	0.9754	0.9465	0.9558
Figure 7x	0.8453	0.9388	0.8430	0.8757	0.8436	0.9499	0.8398	0.8778	0.8440	0.9510	0.8782	0.8782

. Experimental results in Table 2 demonstrate that the performance of APDST filters is a little better than that of the APIDCT with the size of 7 × 7, and the SSIM of G component of APDST is the highest among these three methods.

In addition, to fully demonstrate the experimental results, Figure 8 and Figure 9 show the subjective effects by partially magnifying a part of the captured images. From the images in Figure 8 and Figure 9, it is difficult to distinguish that which method can obtain better performance by naked eyes. It also means that the similar performance in subjective aspect can be obtained by these three methods.

5. Conclusions

Based on in-depth research on APDF and WAPDF, this paper unifies the construction methods of APDF. The features of the WAPBT are analyzed first, and the theory that the WAPBT with rectangular windows is exactly the conventional APBT is concluded. Thus, WAPBT is a generalized form of APBT. The unity of the definition of APT is also given in this paper. The interpolation filters based on DST are applied to image demosaicking. The filter with a size of 7 × 7 is used to interpolate the G component, while the R and B components are interpolated using a rotated filter with a size of 11 × 11. The interpolated image of the proposed scheme has good visual performance and is similar to that of other two methods in subjective aspect. Compared with other two methods, experimental results show that although APDF has more computational complexity than bilinear interpolation, it, including the APIDCT filter and APDST filter, can achieve better quality reconstructed images in objective aspect.

In this paper, the constructed low-pass filter is used in image demosaicking to test its interpolation performance. As the topics of future research, more low-pass filters will be constructed and applied to image interpolation, image denoising [21], image resizing [22], and many other fields of image processing.