Analysis of MTF in TDI-CCD Subpixel Dynamic Super-Resolution Imaging by Beam Splitter

Abstract

The subpixel dynamic imaging technique of a beam splitter is one of the most effective super-resolution imaging methods. Aiming to create a linear time delay integration charge coupled device (TDI–CCD) subpixel imaging system based on the optical assembly method, its modulation transfer function (MTF) is analyzed based on the spatial over-sampling theory. Firstly, Fourier transformation of the sampling point is used to describe the frequency domain characteristics of TDI–CCD, which transform a unit cell of the spatial sampling lattice into a bandwidth cell in the spatial–frequency domain. Considering the effects of velocity mismatch and misalignment, the best subpixel staggering position of the linear TDI–CCD pair is given. Moreover, according to the analysis of the MTF of super-resolution reconstruction results from multiple subpixel images with random spatial offsets, the condition of sampling in the limitation of the enhancement of MTF is obtained. The numerical simulation and real experimental analysis reveal results that are consistent with the theoretical model.

1. Introduction

Obtaining high resolution is an important goal of optical satellite remote-sensing technology. At present, the second-generation transmission type of remote sensing satellite is equipped with discrete imaging sensors, such as time delay integration charge coupled device (TDI–CCD). However, the under-sampling effect caused by the pixel size and sampling interval restricts the resolution of the image according to Shannon sampling theorem [1]. Although the resolution of the image can be improved by improving the area and spatial density of the optical sensor or using various complex optical systems such as free surface design [2,3], the expense is excessive.

The technique of subpixel dynamic super-resolution imaging can overcome the under-sampling effect partially without changing new charge coupled device (CCDs) [4,5]. It use a set of undersampled (aliased) low-resolution (LR) images with subpixel shift to reconstruct a high-resolution (HR) image [6,7], namely oversampling operation (Figure 1). It aims to reconstruct the high-frequency contents which are commonly confused with the low-frequency contents in the LR images accessed by oversampling operation [8]. There have been proposed many subpixel imaging systems according to the oversampling operation formats. For example, the Canadian Defense Research Organization developed a high-performance micro-scanning imaging system for the application of infrared focal plane devices [9], which is a type of technique that doubles the resolution of the staring array imaging device. This technique can acquire multi-frame images of the same scene and detect the phase shift at each time. The French National Center for Space Research (CNES) applied the technique of subpixel imaging to Systeme Probatoire d’Observation dela Tarre 5 (SPOT5) satellites [10], which resulted in an increase in imaging resolution from 5 m to 3 m. The hot spot recognition system (HSRS) [11] sensor, which was developed by German Aerospace Center, and ADS40 [12] digital aerial remote-sensing camera, which was developed by the Lycra company, have adopted the similar technique of subpixel imaging. There are lots of analyses that come forth to evaluate the performance of such subpixel imaging methods. Wang discussed qualitatively the effect of micro-scanning modes on the quality of the image [13,14,15]. Zhang [16] introduced a super-resolution imaging system with oversampling technique. Hadar [17] analyzed the sampling performance of the CCD. However, this study did not mention the conditions under which the CCD reaches its sampling limit, and there was no analysis of the super-resolution reconstruction performance for multiple subpixel images.

Liu [18] proposed another subpixel imaging technique (Figure 2) using a beam splitter to divide the light into two orthotropic beams. A pair of linear array CCD detectors are set up in the two imaging plane positions around the beam splitter with the staggered subpixel-level distance in the linear array direction. So a pair of conjugate images (called a subpixel image pair) would be simultaneously got by the CCD pair via this linear push-broom imaging system for the moving scenes. Through super-resolution processing, a new image with higher spatial resolution in the linear array direction would be constructed [19,20,21].

However, Liu’s work [18] only focused on the CCD sensors. Our subsequent research has updated the subpixel imaging system with TDI-CCDs. Considering MTF is the popular evaluating indicator for the imaging system, this paper proposes MTF analysis method based on discrete spatial oversampling for TDI–CCD subpixel imaging system. In Section 2, the improvement performance of MTF under the over-sampling conditions of subpixel dynamic super-resolution imaging is analyzed for CCD and TDI-CCD linear array in turn. In addition, the conditions of sampling in the limitations of the enhancement of MTF based on both linear CCD and TDI–CCD are given. These two cases are simulated and verified in Section 3. Finally, concluding remarks are presented in Section 4.

2. Sampling MTF of Subpixel Dynamic Super-Resolution Imaging

2.1. Partition Method of CCD

The most important common feature of the crystal’s inner structure is the periodicity of the molecules (atoms and ions) in the space [22]. The Wigner–Seitz cell is the smallest periodic unit of lattice, which is selected according to the following method: in the space lattice, take any node as the origin point; select all nodes near the origin point (if necessary, then consider the second nearest neighbor) and then draw the perpendicular bisectors of these lines [23] to form a rectangle spatial unit surrounded by these perpendicular bisectors (Figure 3) [24,25,26].

Considering the similarity characteristics between the inherent periodicity of CCD pixel spatial distribution and the Wigner–Seitz lattice (Figure 3), the lattice division theory can be used to describe the spatial sampling function of CCD with the traditional square shape (Figure 3a) or the other irregular shapes (Figure 3b). Solid line boxes are regarded as Wigner-Seitz lattice.

Setting the center distance of the pixel in every CCD as p, the pixel size is c, due to now most of the CCD duty cycles tend to 100% because of the use of microlens, c/p = 1. Therefore, we defined the Wigner–Seitz function $w(x)$, normalized with respect to the area of the Wigner–Seitz (WS) lattice as the following:

$$w(x)=\{\begin{array}{l}\frac{1}{p}\times \frac{c}{p}=\frac{1}{p},\mathrm{for}x\mathrm{inside}\mathrm{the}\mathrm{WS}\mathrm{cell}\\ 0,\mathrm{for}x\mathrm{outside}\mathrm{the}\mathrm{WS}\mathrm{cell}\end{array}$$

(1)

2.2. MTF of Over-Sampling with Linear CCDs

Assuming the linear CCD pixel size is p and its duty cycle is 100%, the over-sampling result of CCD pair in Figure 2 with the staggered offset a can be shown in Figure 4c.

We selected the center of each CCD pixel as the coordinate origin to establish a coordinate system. For the results from the new sampling, the Wigner–Seitz lattice is asymmetric. The intervals of Wigner–Seitz lattice are: $-\left(p-a\right)/2~a/2;-a/2~\left(p-a\right)/2$.

The MTF of the sampling results is (The derivation details can be found in Appendix A):

$$MT{F}_2=\sin c(\pi fp)\{\frac{p-a}{p}\sin c[\pi f(p-a)]+\frac{a}{p}\sin c(\pi fa)\}$$

(2)

For obtaining the condition of limitation, we set $\frac{d(MT{F}_2)}{da}=0$, the result is as below:

$$\cos [{\pi }^{2}f(p-a)]=\cos ({\pi }^{2}fa)$$

(3)

As Equation (3) can be applied to all types of $f$, when $p-a=a$ and $a=p/2$, $MT{F}_2$ may has the maximal value.

According to the above analysis, assuming that there are n pair of linear CCDs participated in the subpixel imaging with the staggered offset values of $a_i(i=1,2,\cdots ,2n-1)$ respectively, the over-sampling MTF can reach its maximal value $MT{F}_{mL}$ when $a_1=a_2=\cdots =a_{2n-1}=p/2n$. (The detailed derivation is shown in Appendix A).

2.3. MTF of Over-Sampling with TDI–CCD

Comparing with CCD, the output signal of TDI-CCD pixel is the accumulation from multiple CCD linear arrays for the same scene during the scanning process to improve the response signal-noise rate (SNR) [27]. Actually, there often exists the drift angle caused by the influence of the space-borne camera. Let $V_g$ represent the moving speed of the target (or the push-broom canning direction of camera), the angle between $V_g$ and TDI integral direction is the drift angle $\theta$:

$$\theta =\arctan \left(\frac{V_r}{V_c}\right)$$

(4)

where $V_r$ is the velocity component of $V_g$ in the direction of the CCD linear array; and $V_c$ is the velocity component of $V_g$ in the TDI integral direction (Figure 5b).

If the TDI–CCD line scanning speed and the target speed of motion is not synchronized, which may cause the image shift along the drift angle $\theta$. Therefore, the output response of the pixel is:

$$\begin{array}{l}p(x,y)=\left[\frac{1}{pM\tan \theta }\cdot rect(\frac{x}{pM\tan \theta })\right]\ast \left[\frac{1}{pMd}\cdot rect(\frac{y}{pMd})\right]\\ =\{\begin{array}{l}\frac{1}{p^2{M}^2\tan \theta \cdot d}\hspace{1em}\left|x\right|<\frac{pM\tan \theta }{2},\left|y\right|<\frac{pMd}{2}\\ 0\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\mathrm{elsewhere}\end{array}\end{array}$$

(5)

where pixel’s center distance of the TDI–CCD is p (the length is same as the width); M is the TDI integral stages; $d=\frac{\Delta V}{V_i}$ is the synchronization accuracy; $\Delta V$ a is the difference between the image scanning speed $V_i$ and the TDI charge transfer rate.

The Fourier transform of $p(x,y)$ is:

$$P(u,v)=\sin c(\pi u\cdot pM\tan \theta )\cdot \sin c(\pi v\cdot pMd)$$

(6)

For oversampling in the imaging of TDI–CCD, the shifted distance is b in the vertical distance, assuming that the shifted distance is a in the horizontal direction (Figure 6).

A square and a triangle in Figure 6a are regarded as a basic unit, with the dashed box shown in the lower right corner. Using the division method of Wigner–Seitz cell interpolation, the regular square’s lattices are obtained and shown in Figure 6b.

The Wigner–Seitz function of TDI–CCD $w(x,y)$ is equal to $\frac{4}{p^2}$ when $x\in (-\frac{p-a}{2},\frac{a}{2})$ and $y\in (-\frac{p-b}{2},\frac{b}{2})$. In other cases, it is 0.

The modulation transfer function (MTF₂) of the sampling results from two TDI-CCD is (The derivation details can be found in Appendix A):

$$\begin{array}{ll}MT{F}_2=& \sin c(\pi u\cdot pM\tan \theta )\cdot \{\frac{a}{p}\sin c(\pi ua)+(\frac{p-a}{p})\sin c[\pi u(p-a)]\}\\ & \cdot \sin c(\pi v\cdot pMd)\cdot \{\frac{b}{p}\sin c(\pi vb)+(\frac{p-b}{p})\sin c[\pi v(p-b)]\}\end{array}$$

(7)

For obtaining the condition of limitation, setting $\frac{\partial (MT{F}_2)}{\partial a}=0$, we can obtain the following:

$$\cos [\pi u(p-a)]=\cos (\pi ua)$$

(8)

Then setting $\frac{\partial (MT{F}_2)}{\partial b}=0$, thus we can obtain:

$$\cos [\pi v(p-b)]=\cos (\pi vb)$$

(9)

As Equations (8) and (9) are applicable to all of $u$ and $v$, when $p-a=a$ and $p-b=b$, the MTF₂(a, b) has a maximum value when $a=\frac{p}{2}$ and $b=\frac{p}{2}$.

Similar to analysis in Section 2.2, when there are n image pairs generated from TDI–CCD with shift distance $a_i$ and $b_i$$(i=1,2,\cdots ,2n-1)$ respectively. From the derivation process in Appendix A, when $a_1=a_2=\cdots =a_{2n-1}=p/2n$ and $b_1=b_2=\cdots b_{2n-1}=p/2n$, $MT{F}_{mT}$ reach its maximum value.

3. Experiments

We have verified our analysis result above via mathematical simulation and subpixel image pairs from the real imaging system.

3.1. Simulation Analysis

3.1.1. Simulation of Subpixel Imaging with Linear CCD

The MTF simulation results according to Section 2.2 with the normalized pixel size p are shown in Figure 7.

The blue curve represents the sampling MTF of the linear CCD pair with the ideal staggered distance $a=\frac{p}{2}$. The green curve represents the sampling MTF with $a=\frac{p}{4}$. The red curve is the sampling MTF from a single linear CCD.

From Figure 7, we can see the MTF value at Nyquist frequency are 0.5732, 0.5388 and 0.4053 respectively. So, subpixel imaging apparently does improve MTF when the linear CCD pair are staggerd with ideal distance of half a pixel size, which is consistent with theoretical result in Section 2.2.

In order to quantitatively evaluate the influence due to the misaligned sampling points on the MTF in subpixel imaging, we will use the parameter RM which is the ratio of $MT{F}_a$ to $MT{F}_i$ under the Nyquist frequency as the basis for the evaluation,

$$RM=\frac{MT{F}_a}{MT{F}_i}$$

(10)

where $MT{F}_a$ represents the MTF of the sampling point at any position a, which $a\in [0,p]$, $MT{F}_i$ represents the ideal MTF.

From Figure 8, we can see obviously that a = p/2 is the ideal position, where MTF has the maxim value and RM = 1. The more the distance of a deviates from the ideal location, the lower the RM value.

Figure 9 shows the value of MTF at the Nyquist frequency in the subpixel imaging system with multiple linear CCDs that satisfy the ideal sampling condition a_i = p/2n according to result in Section 2.2. MTF appears an exponential growth trend with the imaging pairs number of linear CCDs. So, if we want to get a high resolution image through sub-pixel dynamic super-resolution imaging, 3 or 4 subpixel image pairs may be the most appropriate choice.

3.1.2. Simulation of Subpixel Imaging with Linear TDI–CCD

As using TDI will increase the effect of a mismatch in speed, the actual drift angle $\theta$ is very small (0–0.3°) when $M\cdot d\le 2$. In order to facilitate simulation, we set $M\cdot d=2$ and θ = 0.15°. Some other parameters are set according to ZY-3 satellite nadir camera in

Table 1. Key technical parameters of the ZY-3 satellite nadir camera.

Parameter	Index
Ground resolution	2.1 m
Width	51 km
MTF of Lab	0.23 (71.5 lp/mm)
SNR	$\ge 25$
Pixel size	7 μm
TDI stages	24
CCD duty cyle	100%

. The MTF can draw the surface with changes in u and v according to results in Section 2.3, as shown in Figure 10.

Figure 10a shows the MTF of single TDI-CCD, Figure 10b shows the TDI-CCD sampling points in the best position when $a=b=\frac{p}{2}$, Figure 10c shows the result with the serious offset of $a=b=\frac{14p}{15}$, Figure 10d shows the result with a minor offset of $a=b=\frac{2p}{3}$. It can be seen that the MTF at Nyquist frequency in the best position is significantly better than the other two graphs from these results. To highlight their differences, Figure 11a,b give the differences between MTF in Figure 10b,c, MTF in Figure 10b,d respectively. We can suppose that if subpixel imaging technology were applied in ZY-3, the resolution will be increased by almost 2 times at best under the condition of accurate imaging position. Although this is only a theoretical simulation, the resolution should also be clearly improved in the application of remote sensing cameras.

3.2. Verification of Real Subpixel Imaging System

We have realized a subpixel camera according to Figure 2 using IL-E2 TDI-CCDs (Figure 12). The

Table 2. Main indexes of camera and TDI-CCD.

Parameter	Index
CCD type	IL-E2 TDI-CCD
Number of pixels	2048
Dynamic range	1600:1
Pixel size	13 μm
TDI stages	24
Speed of valid data	30 MHz

gives the main indexes of camera and TDI-CCD. TDI–CCD line scanning speed and the target speed of motion is synchronized in this subpixel imaging.

We used WT1005-62 transmitted resolution board to examine the MTF of this subpixel imaging system. Figure 13b reveals the reconstruction result of the image pair which has an inexact offset of half a pixel based on projections onto convex sets (POCS) algorithm. Figure 13c shows the same processing result of the image pair with the exact offset of half a pixel.

Figure 14 demonstrates more details with the 1:1 view. We found that the brightness contrast of Figure 14b is still better than Figure 14a, although the contour information is still indistinct. The resolution of Figure 14c is clearly improved, especially in the left second and fourth pairs of lines in the first row. There are many gray and indistinct regions in Figure 14d, with Figure 14e being clearer than the origin image despite the continuous presence of vague edges. In Figure 14f, each pair of lines can be clearly distinguished, especially those in the red box and in 45° direction. The experimental results show that the image quality of two images when taken in the ideal position is better than in the non-ideal position. A half pixel is the best imaging offset between two images. At the same time, this result is supported by the simulation results from the comparison of MTF (Figure 15).

4. Conclusions

In this paper, we analyzed the MTF in TDI–CCD subpixel dynamic super-resolution imaging using a beam splitter. Firstly, we established the oversampling MTF calculation model for the imaging system using linear CCD with the subpixel-level staggered distance in the CCD array direction. Following this, the oversampling MTF calculation model for the subpixel imaging system of TDI–CCD is deduced by adding the subpixel-level staggered distance in the push-broom scanning direction. Furthermore, constraints due to the CCD sampling limit in the MTF enhancement were given. In fact, the MTF calculation model proposed in this paper can be applied to other sub-pixel imaging systems. Finally, the experimental results were relatively consistent with the theory, which proves that this theory can provide a mathematical basis for the super-resolution reconstruction process. It points to future directions for the design standards of high resolution remote sensing cameras.