Trends in Correlation-Based Pattern Recognition and Tracking in Forward-Looking Infrared Imagery

Abstract

In this paper, we review the recent trends and advancements on correlation-based pattern recognition and tracking in forward-looking infrared (FLIR) imagery. In particular, we discuss matched filter-based correlation techniques for target detection and tracking which are widely used for various real time applications. We analyze and present test results involving recently reported matched filters such as the maximum average correlation height (MACH) filter and its variants, and distance classifier correlation filter (DCCF) and its variants. Test results are presented for both single/multiple target detection and tracking using various real-life FLIR image sequences.

1. Introduction

Pattern recognition deals with the detection and identification of a desired pattern or target in an unknown input scene, which may or may not contain the target, and the determination of the spatial location of any target present. In pattern recognition or classification, the input is an image while the output is a decision signal based on some characteristic features of the input image. The number of features is usually fewer than the total necessary to describe the complete target of interest, and this leads to a loss of information. Because of the crucial role of decision making required in pattern recognition, it is fundamentally an information reduction process, whereby, it is not possible to reconstruct the pattern but it is possible to give a precise decision [1–3].

Although a great deal of effort has been expended on detecting objects in visual images, only limited amount of work has been reported on the detection and tracking of targets in infrared images. In general, existing methods on infrared images work for a limited number of situations due to various practical constraints. An infrared sensor detects infrared radiation and converts it to an image by converting the temperature difference between an object and the surrounding background. The temperature scale is converted into a color scale or a gray scale on a display and in this way an image is obtained. This type of sensor or camera can image an object through smoke in a burning house, heat leaking from a house or objects in the absence of any reflected light (at night) [4]. The images captured by infrared sensors are becoming an integral part of the ongoing research on automatic target recognition (ATR).

Forward-looking infrared (FLIR) images are frequently used in ATR applications. The detection and discrimination of targets in infrared imagery has been a challenging problem due to low signal-to-noise ratio (SNR) and the variability of target and clutter signatures. The FLIR sequences, tested in this work, are recorded from a moving platform and include independently moving objects under various distortions and background variations. Thus, sensor ego-motion and object motions induce coupled motions into the FLIR images, which make the detection and tracking of the objects extremely complicated. To detect independently moving objects in FLIR image sequences, the sensor properties must also be taken into account.

Real life FLIR imagery demonstrates a number of well-known challenges such as significantly high level of variability of target thermal signatures, size/aspect, locations within the scene; large number of target classes; lack of prior information; obscured targets; competing cluttered background scenery; different geographic, meteorological and weather conditions; time of the day; high ego-motion; sensor noise; and variations caused by translation, rotation, and scaling of the targets. Furthermore, inconsistencies in the signature of targets, similarities between the signatures of different targets, limited training and testing data, camouflaged targets, non-repeatability of target signatures, and difficulty in exploiting contextual information make the recognition problem even more challenging in target detection in FLIR imagery. In the case of FLIR images, additional challenges are caused due to the following important differences [5–7] with visual sequences:

• The thermal images are obtained by sensing the radiation in the infrared spectrum, which is either emitted or reflected by the object in the scene. Due to this property the images obtained from an infrared sensor have extremely low SNR, which results in limited information for performing detection or tracking task.

• FLIR imagery smoothes out object edges and corners leading to a reduction of distinct features.

• The generation and maintenance of kinetic energy usually heats up a moving object (e.g., friction, engine combustion). Consequently, moving objects often appear brighter than the background.

• FLIR images are noisy and have less contrast. Moreover, they often contain dirt on the lens, or local sensor failure at certain pixel locations.

• FLIR sequences are not easily available (especially not from controlled experiments) and have a lower resolution. The sequences available to us are 128 × 128 pixels as compared to the 512 × 512 pixels, and more, of standard visual cameras.

• FLIR sequences are often under difficult circumstances and may have abrupt discontinuities in motion.

Due to the limitations and difficulties of FLIR imagery, they demand more robust techniques than visual sequences. This paper presents some widely used pattern recognition and target tracking techniques adopted for FLIR imagery. In this paper, we discuss several target detection and tracking algorithms which are based on the recently reported matched filter-based correlation techniques such as the MACH, EMACH, DCCF, and PDCCF filters. The performance of these algorithms was tested using real life FLIR image sequences supplied by the Army Missile Command.

2. Matched Filter-Based Correlation

The matched filter-based correlator was first introduced in 1964 [8]. In this technique, the input signal f(σ,ε) is Fourier transformed to yield

$$F(o,\upsilon )=\Im [f(\sigma ,\varepsilon )]$$

(1)

where ℑ represents the Fourier transform operation, σ and ε represents the spatial domain variables, and σ and υ represents the frequency domain variables, respectively. The correlation output is obtained by inverse Fourier transform operation given by

$$g(\sigma ,\varepsilon )={\Im }^{-1}[F(o,\upsilon )H(o,\upsilon )]$$

(2)

where, ℑ ⁻¹ represents 2D inverse Fourier transform operation.

If the input object is moved laterally in the input plane, the Fourier transform remains fixed in space but is multiplied by a phase factor that depends on the lateral movement. Therefore, the coordinates of the bright correlation output is proportional to the coordinates of the signal f(σ,ε) located at the input plane. The intensity of the bright correlation spot is proportional to the degree to which the input and the filter functions are matched. This is also valid for multiple objects present at different locations of the input plane. This correlation system provides a great deal of sensitivity since it is both phase matched and amplitude matched [9–11].

For the complex matched filter (CMF), the frequency plane filter function is expressed by

$$H_{\text{cmf}}=R^{\ast }(o,\upsilon )=|R(o,\upsilon )|exp[-j\varphi (o,\upsilon )]$$

(3)

where, |R(ο,υ)| is the amplitude and φ(ο,υ) is the phase factor of the Fourier spectrum of the reference function r(σ,ε). When the input is similar to r(σ,ε), the phase variation is canceled at the Fourier plane, thus producing a plane wave of light. The correlation peak corresponding to CMF is not very sharp due to the squaring of the magnitude of r(σ,ε). Consequently, the resulting diffraction efficiency of a CMF is very poor. This filter is also unacceptably sensitive to even small changes in the reference signal or image. Currently available spatial light modulators (SLMs) cannot accommodate the full complex frequency response needed by CMFs.

3. Phase Only Filter (POF)

The optimum case with respect to light efficiency for a matched filter is realized by a phase only filter (POF) structure [10]. This filter is obtained by omitting the amplitude information and is defined as

$$H_{\text{pof}}(o,\upsilon )=\frac{R^{\ast }(o,\upsilon )}{\left|R(o,\upsilon )\right|}=exp[-j\varphi (o,\upsilon )]$$

(4)

Besides the improvement in light efficiency, the correlation peak intensity is also enhanced with a POF. However, although the autocorrelation peak intensity is higher than that of a CMF, it is not as sharp as might be produced by an IF since the product H_pof(ο,υ)R(ο,υ) is not necessarily a constant.

4. Amplitude-Modulated Phase Only Filter (AMPOF)

Amplitude modulated phase only filter (AMPOF) [12] is given by

$$H_{\text{ampof}}(o,\upsilon )=\frac{A}{B+\left|R(o,\upsilon )\right|}exp[-j\varphi (o,\upsilon )]$$

(5)

where A and B are either constants or functions of ο and υ. The gain control factor A guarantees that the transmittance of the filter is less than unity. With the inclusion of B; the pole problem is solved and at the same time it is possible to yield very high autocorrelation peak.

5. Synthetic Discriminant Functions (SDF)

The SDF-based correlation filters had shown robust performance for distortion tolerant pattern recognition [13–15]. Assume x₁(σ,ε), x₂(σ,ε), …, x_N(σ,ε) denote N training images representing possible distortions to a reference image x(σ,ε). The 2D Fourier transform of x(σ,ε) may be expressed as

$$X(o,\upsilon )=\int \int x(\sigma ,\varepsilon )exp[-j2\pi (o\sigma +\upsilon \varepsilon )]\times \text{d}\sigma \text{d}\varepsilon$$

(6)

A composite image h(σ,ε) is designed from the training images such that when the complex conjugate of its Fourier transform, denoted as H*(ο,υ), is correlated with input Fourier transform, a similar output is obtained for all N inputs, x₁(σ,ε), x₂(σ,ε), …, x_N(σ,ε). This type of correlator is known as frequency plane correlator. For this filter, the resulting spatial domain correlation output c(τ_σ,τ_ε) may be expressed as [13]

$$\begin{array}{ll}c({\tau }_{\sigma },{\tau }_{\varepsilon })& =\int \int F(o,\upsilon )H^{\ast }(o,\upsilon )exp[j2\pi (o{\tau }_{\sigma }+\upsilon {\tau }_{\varepsilon })]\text{d}o\text{d}\upsilon \\ & =\int \int h^{\ast }(\sigma ,\varepsilon )f(\sigma +{\tau }_{\sigma },\varepsilon +{\tau }_{\varepsilon })\text{d}\sigma \text{d}\varepsilon \\ & =h(\sigma ,\varepsilon )\odot f(\sigma ,\varepsilon )\end{array}$$

(7)

where ⊙ denotes a two-dimensional cross-correlation operation.

In the Equal Correlation Peak SDF (ECP-SDF) design, the objective is to select a filter impulse response h(σ,ε) such that the resulting crosscorrelations with all the N training images are the same, which is impossible in practice. Hester and Casasent [13] introduced a technique that requires that only the values at the origin of these crosscorrelations should be the same as shown in the following equation,

$$\begin{array}{cc}{h(\sigma ,\varepsilon )\odot x_i(\sigma ,\varepsilon )|}_{{\tau }_{\sigma }=0,{\tau }_{\varepsilon }=0}& =\int \int h^{\ast }(\sigma ,\varepsilon )x_i(\sigma ,\varepsilon )\text{d}\sigma \text{d}\varepsilon \\ & \begin{array}{cc}=c,& i=1,2,\dots \dots .N\end{array}\end{array}$$

(8)

where c is a prespecified constant. Equation (8) shows that a h(σ,ε) would yield the same constant value c at the origin (location of the autocorrelation peak) for all N training images (i.e., x₁(σ,ε) to x_N(σ,ε)). When the input is a non-training image from the same class, the crosscorrelation output at the origin will be similar to this constant c and it can be recognized. The success of this approach depends on selecting a proper training set.

Assume that h(σ,ε) is a linear combination of the N training images, given by

$$h(\sigma ,\varepsilon )=a_1{x}_1(\sigma ,\varepsilon )+\dots \dots \dots \dots \mathrm{..}{a}_N{x}_N(\sigma ,\varepsilon )$$

(9)

where the coefficients a₁, a₂, … , a_N are determined in a way to satisfy the constraints of Equation (8). Substituting Equation (9) into Equation (8), we get

$$\sum _{i=1}^N{a}_i^{\ast }{R}_{ij}=c,j=1,2,\dots \dots ,N,$$

(10)

where

$$R_{ij}=\iint x_i^{\ast }(\sigma ,\varepsilon )x_j(\sigma ,\varepsilon )\text{d}\sigma \text{d}\varepsilon$$

(11)

is the inner product, i.e., the crosscorrelation at the origin of the training images x_i(σ,ε) and x_j(σ,ε). If the training images are real, then there is no need for the conjugate operation shown in Equation (11). Equation (10) represents N complex linear equations with N complex unknowns, a₁, a₂, …, a_N, respectively. These equations can be solved by using any standard techniques, such as Gauss-Seidel Elimination method.

6. Modified Synthetic Discriminant Functions

The ECP-SDF is designed to produce a value c_i at the origin of the output plane when the i-th training image is used as the input. However, there are some practical problems in using this filter for pattern recognition applications [16–18]. Consequently, some improvements on ECP_SDF are suggested which are introduced in the following subsections.

6.1. Generalized SDF

Assume that the training images x₁(σ,ε), x₂(σ,ε), …, x_N(σ,ε) are sampled to yield arrays x₁(m,n), x₂(m,n), …, x_N(m,n) each with ρ = ρ₁ρ₂ pixels, where ρ₁ is the number of pixels in the vertical direction while ρ₂ is the number of pixels in the horizontal direction of each image. It is also assumed that ρ-dimensional column vectors x₁, x₂, …, x_N are obtained by placing the elements in these training images in vectors where the scanning direction is from left to right and from top to bottom. Similarly, the ρ-dimensional column vector h is used to denote the composite image h(m,n). Then the constraints in Equation (8) can be rewritten as

$$\begin{array}{cc}{\mathbf{\text{h}}}^{+}{\mathbf{\text{x}}}_i=c_i,& i=1,2,3,\dots \dots .N\end{array}$$

(12)

where the superscript ⁺ denotes the conjugate transpose operation. The data matrix X is assumed to have the vector x_i as its i-th column and is thus a ρ × N matrix. It is also assumed that ρ ≫ N, i.e., the number of pixels in the training images is much larger than the number of training images, and that the columns of this matrix are linearly independent. Using this notation, Equation (12) can be rewritten as

$${\mathbf{\text{X}}}^{+}\mathbf{\text{h}}={\mathbf{\text{c}}}^{\ast }$$

(13)

The ECP-SDF assumes that the composite image h is of the following form

$$\mathbf{\text{h}}=\mathbf{\text{Xa}}$$

(14)

where a is the vector of coefficients. Substituting Equation (14) into Equation (13) and solving for a yields

$$\mathbf{\text{a}}={({\mathbf{\text{X}}}^{+}\mathbf{\text{X}})}^{-1}{\mathbf{\text{c}}}^{\ast }$$

(15)

The filter vector h can be obtained by substituting Equation (15) into Equation (14) to get

$${\mathbf{\text{h}}}_{\text{ECP}}=\mathbf{\text{X}}{({\mathbf{\text{X}}}^{+}\mathbf{\text{X}})}^{-1}{\mathbf{\text{c}}}^{\ast }$$

(16)

A general expression for h satisfying Equation (13) is given by

$${\mathbf{\text{h}}}_{\text{GSDF}}=\mathbf{\text{X}}{({\mathbf{\text{X}}}^{+}\mathbf{\text{X}})}^{-1}{\mathbf{\text{c}}}^{\ast }+[{\mathbf{\text{I}}}_d-\mathbf{\text{X}}{({\mathbf{\text{X}}}^{+}\mathbf{\text{X}})}^{-1}{\mathbf{\text{X}}}^{+}]\mathbf{\text{z}}$$

(17)

where I_d is the ρ × ρ diagonal identity matrix and z is any column vector with ρ complex entries. The ECP-SDF is obtained from Equation (17) when z = 0. The filter vector in Equation (17) is known as the generalized SDF [17].

6.2. Minimum Variance SDF

Consider a situation where the input image is one of the training images x_i corrupted by additive noise n. Then the resulting output value y, i.e., the value of the crosscorrelation at the origin is given by

$$\begin{array}{c}y={\mathbf{\text{h}}}^{+}({\mathbf{\text{x}}}_i+\mathbf{\text{n}})\\ =c_i+{\mathbf{\text{h}}}^{+}\mathbf{\text{n}}\end{array}$$

(18)

where h is designed to satisfy Equation (12). From Equation (18) it is evident that the output y is the desired output c_i corrupted by the random variable (h⁺n). The minimum variance synthetic discriminant function (MVSDF) [18] attempts to design h such that the variance in the output caused by input noise is minimized while satisfying the constraints in Equation (13).

Assume that the real noise vector n is a zero-mean vector with a ρ × ρ covariance matrix Σ. The variance of y corresponding to h⁺n can be expressed as

$${\sigma }_y^2=E\left\{{\left|{\mathbf{\text{h}}}^{+}\mathbf{\text{n}}\right|}^2\right\}=E\left\{{\mathbf{\text{h}}}^{+}\mathbf{\text{n}}{\mathbf{\text{n}}}^{+}\mathbf{\text{h}}\right\}={\mathbf{\text{h}}}^{+}\sum \mathbf{\text{h}}$$

(19)

It is desired that ${\sigma }_y^2$ in Equation (19) is as small as possible, which will ensure that the output values are close to the constrained values even in the presence of noise. Minimizing ${\sigma }_y^2$ in Equation (19) subject to the constraints in Equation (13) leads to the following MVSDF [18]

$${\mathbf{\text{h}}}_{\text{MVSDF}}={\sum }^{-1}\mathbf{\text{X}}{({\mathbf{\text{X}}}^{+}{\sum }^{-1}\mathbf{\text{X}})}^{-1}{\mathbf{\text{c}}}^{\ast }$$

(20)

This MVSDF is indeed optimal from noise tolerance considerations. One difficulty in using this MVSDF is that often Σ is not known. Even when it is known, it is impossible to calculate its inversion. Another problem is that the MVSDF controls only one point (the origin) in the output-correlation plane. Thus, large sidelobes may be observed in the correlation output.

6.3. Frequency-Domain SDFs

It is often more convenient to design the filters in the frequency domain [16–18]. Assume f(σ,ε) is the image and H*(ο,υ) is the complex filter function. Then the resulting correlation output c(τ_σ,τ_ε) at the origin is given by

$$\begin{array}{cc}c(0,0)& =\iint H^{\ast }(o,\upsilon )F(o,\upsilon )\text{d}o\text{d}\upsilon \\ & ={\widehat{\mathbf{\text{h}}}}^{\ast }\widehat{\mathbf{\text{f}}}\hfill \end{array}$$

(21)

where ˆ indicates that the corresponding vector or matrix is obtained by sampling frequency domain functions and the superscript ⁺ indicates a conjugate transpose operation. Because c_i(0,0) is constrained to be c_i, i = 1, 2, …, N, the constraints can be rewritten as

$${\widehat{\mathbf{\text{F}}}}^{+}\widehat{\mathbf{\text{h}}}={\mathbf{\text{c}}}^{\ast }$$

(22)

where $\widehat{\mathbf{\text{F}}}$ is a matrix with N columns with the i-th column containing ${\widehat{\mathbf{\text{f}}}}_i$. It is obvious that the Equations (13) and (22) are similar.

6.4. Minimum Average Correlation Energy (MACE) Filter

The correlation filters discussed so far control only one point in the correlation plane. For good location accuracy and discrimination, it is necessary to design filters capable of producing sharp correlation peaks. One such filter is the minimum average correlation energy (MACE) filter [19]. Assume x₁(σ,ε), x₂(σ,ε), …, x_N(σ,ε) denote the N training images and X₁(ο,υ), …, X_N(ο,υ) denote their 2D Fourier transforms, respectively. If H*(ο,υ) denotes the transmittance of the filter function, then the filter may be constructed to satisfy the following condition.

$$\begin{array}{cc}\iint X_i(o,\upsilon )H^{\ast }(o,\upsilon )\text{d}o\text{d}\upsilon =c_i,& i=1,2,\dots \dots ,N\end{array}$$

(23)

In addition, the MACE filter minimizes the average correlation plane energy as shown below.

$$\begin{array}{cc}{E}_{\text{ave}}& =\frac{1}{N}\sum _{i=1}^N\iint {\left|c_i({\tau }_{\sigma },{\tau }_{\varepsilon })\right|}^2\text{d}{\tau }_{\sigma }\text{d}{\tau }_{\varepsilon }\\ & =\frac{1}{N}\sum _{i=1}^N\iint {\left|X_i(o,\upsilon )\right|}^2{\left|H(o,\upsilon )\right|}^2\text{d}o\text{d}\upsilon \end{array}$$

(24)

By minimizing E_ave, it is possible to keep the sidelobes in the correlation plane as low as possible. This is essentially an indirect attempt at reducing the problem of sidelobes. To carry out the minimization of E_ave, the usual vector notation is used. If ${\widehat{\mathbf{\text{x}}}}_i$ denote the ρ-dimensional complex column vector obtained by sampling X_i(ο,υ), then the constraints in Equation (23) can be rewritten as

$${\widehat{\mathbf{\text{X}}}}^{+}\widehat{\mathbf{\text{h}}}={\mathbf{\text{c}}}^{\ast }$$

(25)

where $\widehat{\mathbf{\text{X}}}$ is a ρ × N matrix with ${\widehat{\mathbf{\text{x}}}}_i$ as its i-th column. The E_ave in Equation (24) can be expressed as

$$E_{\text{ave}}={\widehat{\mathbf{\text{h}}}}^{+}\widehat{\mathbf{\text{D}}}\widehat{\mathbf{\text{h}}}$$

(26)

where $\widehat{\mathbf{\text{D}}}$ is a ρ × ρ diagonal matrix. The entries along the diagonal are obtained by averaging |X_i(ο,υ)|², i = 1, 2, …, N, and then scanning the average from left to right and from top to bottom. Minimizing E_ave in Equation (26) subject to the constraints in Equation (25) leads to the following filter

$${\widehat{\mathbf{\text{h}}}}_{\text{MACE}}={\widehat{\mathbf{\text{D}}}}^{-1}\widehat{\mathbf{\text{X}}}{({\widehat{\mathbf{\text{X}}}}^{+}{\widehat{\mathbf{\text{D}}}}^{-1}\widehat{\mathbf{\text{X}}})}^{-1}{\mathbf{\text{c}}}^{\ast }$$

(27)

In many simulation studies, filters designed using this approach produced sharp correlation peaks. However, MACE filters appear to have two drawbacks. The first is that there is no noise tolerance built into these filters. The second is that these filters seem to be more sensitive to intra-class variations. Casasent et al. [20] proposed Gaussian MACE filters to reduce the sensitivity of the MACE filters to intra-class variations. The idea behind Gaussian MACE filters is to reduce the sharpness of the resulting correlation peak and thus improve its noise tolerance. MACE filters appear to be the first set of composite filters that attempt to control the entire correlation plane.

6.5. Minimum Squared Error SDF (MSE-SDF) Filter

This SDF design approach yields better approximation of arbitrary output correlation shapes in the minimum squared error (MSE) sense over the MACE filter and this filter is termed as MSE-SDF [21]. Like MACE, this filter must satisfy the usual SDF constraint of Equation (25). Besides, in MSE-SDF, the filter function H(ο,υ) must make the correlation function c_i approximate a prespecified desired shape t_i, i = 1, 2, …, N. One measure of how well c_i approximates t_i is the average squared error E defined as

$$\begin{array}{cc}E& =\frac{1}{N}\sum _{i=1}^N\iint {\left|t_i({\tau }_{\sigma },{\tau }_{\varepsilon })-c_i({\tau }_{\sigma },{\tau }_{\varepsilon })\right|}^2\text{d}{\tau }_{\sigma }\text{d}{\tau }_{\varepsilon }\\ & =\frac{1}{N}\sum _{i=1}^N\iint {\left|T_i(o,\upsilon )-X_i^{\ast }(o,\upsilon )H(o,\upsilon )\right|}^2\text{d}o\text{d}\upsilon \end{array}$$

(28)

where T_i is the Fourier transform of t_i. If ${\widehat{\mathbf{\text{X}}}}_i^{\mathbf{\text{D}}}$ is obtained by converting the vectors ${\widehat{\mathbf{\text{x}}}}_i$ into diagonal matrices, then the average squared error of Equation (28) becomes

$$E=\left[E_d-{\widehat{\mathbf{\text{h}}}}^{+}\widehat{\mathbf{\text{p}}}-{\widehat{\mathbf{\text{p}}}}^{+}\widehat{\mathbf{\text{h}}}+{\mathbf{\text{h}}}^{+}\widehat{\mathbf{\text{M}}}\widehat{\mathbf{\text{h}}}\right]$$

(29)

where

$$E_d=\frac{1}{N}\sum _{i=1}^N({\mathbf{\text{t}}}_i^{+}{\mathbf{\text{t}}}_i)$$

(30)

$$\widehat{\mathbf{\text{p}}}=\frac{1}{N}\sum _{i=1}^N({\mathbf{\text{X}}}_i^{\mathbf{\text{D}}}{\mathbf{\text{t}}}_i)$$

(31)

$$\widehat{\mathbf{\text{M}}}=\frac{1}{N}\sum _{i=1}^N({\mathbf{\text{X}}}_i^{\mathbf{\text{D}}\ast }{\mathbf{\text{X}}}_i^{\mathbf{\text{D}}})$$

(32)

Minimizing E in Equation (29) subject to the constraints in Equation (25) leads to the following MSE-SDF filter

$${\widehat{\mathbf{\text{h}}}}_{\text{MSE}-\text{SDF}}={\widehat{\mathbf{\text{M}}}}^{-1}\widehat{\mathbf{\text{p}}}+{\widehat{\mathbf{\text{M}}}}^{-1}\widehat{\mathbf{\text{X}}}\left[{({\widehat{\mathbf{\text{F}}}}^{+}{\widehat{\mathbf{\text{M}}}}^{-1}\widehat{\mathbf{\text{F}}})}^{-1}\right]\left[\widehat{\mathbf{\text{c}}}-{\widehat{\mathbf{\text{F}}}}^{+}{\widehat{\mathbf{\text{M}}}}^{-1}\widehat{\mathbf{\text{p}}}\right]$$

(33)

The MSE-SDF filter allows approximating arbitrary correlation shapes rather than zero shape implied in MACE filter design. This explicit control has two benefits [21]. First, correlation shapes can be selected with the linear/nonlinear postprocessing in mind and second, those correlation shapes that lead to better filter design can be used instead of simply minimizing the average correlation energy.

7. Maximum Average Correlation Height (MACH) Filter

The primary objective of the correlation filters is to achieve distortion-tolerant recognition of objects in the presence of clutter. This problem is easier to solve for in-plane rotations and scale changes. However, the prevalent method for handling out-of-plane distortions is to use a training set of representative views of the object. Traditionally, in the design of SDF-type correlation filters, linear constraints are imposed on the training images to yield a known value at specific locations in the correlation plane. However, placing such constraints in the correlation plane satisfies conditions only at isolated points in the image space but does not explicitly control the filter's ability to generalize over the entire domain of the training images. Various filters exhibit different levels of distortion tolerance even with the same training set and constraints.

The MACH filter adopts a statistical approach for filter design [22,23]. In addition to yielding sharp peaks and being computationally simple, this filter offers improved distortion tolerance. The reason lies in the fact that training images are not treated as deterministic representations of the object but as samples of a class whose characteristic parameters should be used in encoding the filter.

It is assumed that the training set consists of N images, and that each image of size ρ₁ × ρ₂ contains ρ = ρ₁ρ₂ pixels. The i-th training image for the target class is denoted by x_i(m,n) in the spatial domain, which is represented in the frequency domain by a ρ × 1 vector x_i, obtained by lexicographically reordering its two-dimensional discrete Fourier transform, X_i(k,l). The Fourier domain filter is denoted by the ρ × 1 vector h. The two-dimensional filter H(k,l) is obtained by rearranging h into a two-dimensional image. In this paper, matrices are denoted by uppercase bold-face and vectors by lowercase bold-face characters. The correlation of the i-th training image and the filter can be expressed in the frequency domain as

$${\mathbf{\text{g}}}_i={\mathbf{\text{X}}}_i\mathbf{\text{h}}$$

(34)

where X_i is a ρ × ρ diagonal matrix containing the elements of x_i. Here, g_i denotes the discrete Fourier transform of the i-th correlation output. The deviation in the shape of the correlation plane with respect to some ideal shape vector f is quantified by the average squared error (ASE), defined as

$$\text{ASE}=\frac{1}{N}\sum _{i=1}^N{({\mathbf{\text{g}}}_i-\mathbf{\text{f}})}^{+}({\mathbf{\text{g}}}_i-\mathbf{\text{f}})$$

(35)

Thus, ASE is a measure of distortion with respect to reference shape f, which can be chosen as desired.

In fact, the shape vector f can be treated as a free parameter in the distortion minimization problem. In the design of MSE-SDF [21], f is specified as Gaussian or ring-like shapes in order to sculpt the correlation surface into these forms. In MACH, the choice of f is such that it causes least variation among the correlation planes and offers minimum ASE. To find the optimum shape f_opt, the gradient of ASE with respect to f is set to zero, given by

$${\nabla }_{\mathbf{\text{f}}}(\mathit{\text{ASE}})=\frac{2}{N}\sum _{i=1}^N({\mathbf{\text{g}}}_i-\mathbf{\text{f}})=0$$

(36)

or,

$${\mathbf{\text{f}}}_{\text{opt}}=\frac{1}{N}\sum _{i=1}^N{\mathbf{\text{g}}}_i=\overline{\mathbf{\text{g}}}$$

(37)

where

$$\overline{\mathbf{\text{g}}}=\frac{1}{N}\sum _{i=1}^N{\mathbf{\text{X}}}_i\mathbf{\text{h}}=\mathbf{\text{Mh}}$$

(38)

is the average correlation plane and $\mathbf{\text{M}}=(1/N){\sum }_{i=1}^N{\mathbf{\text{X}}}_i$ is the average training image expressed as a diagonal matrix. Thus, among all possible reference shapes, the average correlation plane $\overline{\mathbf{\text{g}}}$ offers the smallest possible ASE and the least distortion (in the squared error sense) among the correlation planes.

Substituting $\mathbf{\text{f}}$ in the ASE expression, the average similarity measure (ASM) is obtained as

$$\begin{array}{cc}\text{ASM}& =\frac{1}{N}\sum _{i=1}^N{({\mathbf{\text{g}}}_i-\overline{\mathbf{\text{g}}})}^{+}({\mathbf{\text{g}}}_i-\overline{\mathbf{\text{g}}})\\ & =\frac{1}{N}\sum _{i=1}^N{({\mathbf{\text{X}}}_i\mathbf{\text{h}}-\mathbf{\text{Mh}})}^{+}({\mathbf{\text{X}}}_i\mathbf{\text{h}}-\mathbf{\text{Mh}})\\ & ={\mathbf{\text{h}}}^{+}\left[\sum _{i=1}^N{({\mathbf{\text{X}}}_i-\mathbf{\text{M}})}^{\ast }({\mathbf{\text{X}}}_i-\mathbf{\text{M}})\right]\mathbf{\text{h}}\\ & ={\mathbf{\text{h}}}^{+}{\mathbf{\text{S}}}_x\mathbf{\text{h}}\end{array}$$

(39)

where

$${\mathbf{\text{S}}}_x=\sum _{i=1}^N{({\mathbf{\text{X}}}_i-\mathbf{\text{M}})}^{\ast }({\mathbf{\text{X}}}_i-\mathbf{\text{M}})$$

(40)

is a diagonal matrix measuring the similarity of the training images to the class mean in the frequency domain. For example, if all training images are identical, then S_x would be an all-zero matrix. From Parseval's theorem, it is easy to show that the average squared distance from the correlation planes to their mean is the same as that defined by Equation (39) in the frequency domain [22].

The ASM is one possible metric for distortion since it represents the average deviation of the correlation planes from the mean correlation shape, $\overline{\mathbf{\text{g}}}$. It is also a measure of the compactness of the class. If filter h is viewed as a linear transform, then ASM measures the distances of the training images from the class center under this transform. Minimizing ASM, therefore, leads to a compact set of correlation planes that resemble each other and exhibit the least possible variations. The distortions of the object in the input plane are represented by the training images, x_i. These distortions are reflected in the output as variations in the structure and shape of the corresponding correlation planes, g_i, and are quantified by ASM. If the filter successfully reduces the distortions, then distorted input images should yield similar output planes, leading to a small value of ASM. Conversely, if ASM is minimum and it is well shaped by design, then all true-class correlation planes are expected to resemble $\overline{\mathbf{\text{g}}}$ and to exhibit well-shaped structures.

The MACH filter relaxes the correlation peak constraints and maximizes the peak intensity of the average training image. The peak intensity of the average training image is $|\overline{g}(0,0){|}^2$ expressed as

$${\left|\overline{g}(0,0)\right|}^2={\left|\frac{1}{N}\sum _{i=1}^N{\mathbf{\text{h}}}^{+}{\mathbf{\text{x}}}_i\right|}^2={\left|{\mathbf{\text{h}}}^{+}\mathbf{\text{m}}\right|}^2={\mathbf{\text{h}}}^{+}\mathbf{\text{m}}{\mathbf{\text{m}}}^{+}\mathbf{\text{h}}$$

(41)

where m is the Fourier transform of the average training image expressed as a vector.

Here, it is assumed without the loss of generality that the peak occurs at the origin of the correlation plane.

The smaller the value of ASM, the more invariant the response of the filter is. In other words, if ASM is small, then all true-class correlation planes are expected to resemble $\overline{\mathbf{\text{g}}}$. Therefore, it is required by h to produce high correlation peak with the mean image while making ASM small. In addition, it is also required to obtain some degree of noise tolerance to reduce the output noise variance (ONV). For additive input noise, ONV = h⁺Dh, where D is the diagonal power spectral density matrix [23]. While practical noise may be multiplicative and more complicated than implied by simple additive noise, the simple additive noise model at least provides some robustness. The performance criterion used to optimize the MACH filter may be expressed as

$$\begin{array}{cc}J(\mathbf{\text{h}})& =\frac{{(\text{Average peak height})}^2}{\text{ASM}+\text{ONV}}\\ & =\frac{{\left|\overline{g}(0,0)\right|}^2}{\text{ASM}+\text{ONV}}\\ & =\frac{{\left|{\mathbf{\text{h}}}^{+}\mathbf{\text{m}}\right|}^2}{{\mathbf{\text{h}}}^{+}\mathbf{\text{Sh}}+{\mathbf{\text{h}}}^{+}\mathbf{\text{Dh}}}\\ & =\frac{{\mathbf{\text{h}}}^{+}\mathbf{\text{m}}{\mathbf{\text{m}}}^{+}\mathbf{\text{h}}}{{\mathbf{\text{h}}}^{+}(\mathbf{\text{S}}+\mathbf{\text{D}})\mathbf{\text{h}}}\end{array}$$

(42)

The optimum solution is found by setting the derivative of J(h) in Equation (42) with respect to h to zero and is given by [22,23]

$$\mathbf{\text{h}}={(\mathbf{\text{S}}+\mathbf{\text{D}})}^{-1}\mathbf{\text{m}}$$

(43)

The filter in Equation (43) is referred to as the MACH filter because it maximizes the height of the mean correlation peak relative to the expected distortions. For cases where an estimate of D is not available, the white noise covariance matrix is substituted for D, i.e., D = σ²I, where I is a diagonal identity matrix. Hence the simplified MACH filter becomes

$$\mathbf{\text{h}}={(\mathbf{\text{S}}+{\sigma }^2\mathbf{\text{I}})}^{-1}\mathbf{\text{m}}$$

(44)

Replacing σ² by another constant γ, the filter equation becomes

$$\mathbf{\text{h}}={(\mathbf{\text{S}}+\gamma \mathbf{\text{I}})}^{-1}\mathbf{\text{m}}$$

(45)

The robustness of the MACH filter is attributed to the inclusion of the ASM criterion, which reduces the filter sensitivity to distortions, and to the removal of hard constraints on the peak. The later fact enables the correlation planes to adjust to suitable values for optimizing the performance criterion. However, MACH filter can handle those distortions that are well represented in the training set.

The Fourier domain MACH filter obtained in Equation (45) may be converted to the 2D shape or size of the input training images expressed as H(k,l). A 2D test image z(m,n) is Fourier transformed to obtain Z(k,l) which is then correlated with the 2D filter in the Fourier domain using the expression

$$G(k,l)=Z(k,l)H^{\ast }(k,l)$$

(46)

The output spatial domain correlation is obtained by applying the inverse Fourier transform operation to Equation (46) and recording the intensity, given by

$$g(m,n)={\left|{\Im }^{-1}\left[G(k,l)\right]\right|}^2$$

(47)

8. Extended MACH (EMACH) Filter

The average training image used in the MACH filter design is good in representing the average behavior of the desired class, but it fails to capture the finer details of the desired class [24,25]. In fact, the average of training images sometimes looks like a clutter image. Thus, the MACH filter may be inadequate in discriminating the desired class from the clutter, leading to increased false alarm rate. The extended MACH (EMACH) filter is aimed at improving this clutter rejection capability.

The MACH filter is designed to maximize the intensity of the average correlation output at the origin due to training images. The average of correlation peaks is the correlation output due to the average training image. It also maximizes the similarity between the average training image correlation output and those outputs due to all training images from the desired class. Thus, the MACH filter forces all images from the desired class to follow the behavior of the average training image from that class. The MACH filter relies heavily on the mean training image. It amplifies the high-energy (usually low-frequency) components, and at the same time, attenuates the low-energy (usually high-frequency) components of the training set. Thus, by using the mean image m as the only example that represents all training images, a filter may be obtained that does not capture the finer details of the training images. Therefore, this filter may fail to discriminate the desired class from the clutter. The MACH filters possess attributes that may lead to detect clutter images as targets. One such attribute is that all training images follow the same behavior as the average training image. However, the average training image is not necessarily a good representative of the desired class.

To control the relative contribution of the desired class training images as well as their average, a new metric, called all image correlation height (AICH) [25], is introduced and defined as

$$\begin{array}{cc}\text{AICH}& =\frac{1}{N}\sum _{i=1}^N{\left[{\mathbf{\text{h}}}^{+}\left({\mathbf{\text{x}}}_i-\beta \mathbf{\text{m}}\right)\right]}^2\\ & =\frac{1}{N}\sum _{i=1}^N{\left({\mathbf{\text{h}}}^{+}{\mathbf{\text{x}}}_i-\beta {\mathbf{\text{h}}}^{+}\mathbf{\text{m}}\right)}^2\end{array}$$

(48)

where β is a parameter that takes a value between 0 and 1 and governs the relative significance of the average training image in the filter design. By controlling β, the designed filter is prevented from being overwhelmed by the biased treatment of the low-frequency components represented by the average image. Here, the AICH must be optimized and to be able to do that, Equation (48) may be rewritten as

$$\begin{array}{cc}\text{AICH}& =\frac{1}{N}\sum _{i=1}^N\left({\mathbf{\text{h}}}^{+}{\mathbf{\text{x}}}_i-\beta {\mathbf{\text{h}}}^{+}\mathbf{\text{m}}\right){\left({\mathbf{\text{h}}}^{+}{\mathbf{\text{x}}}_i-\beta {\mathbf{\text{h}}}^{+}\mathbf{\text{m}}\right)}^{+}\\ & ={\mathbf{\text{h}}}^{+}\left[\frac{1}{N}\sum _{i=1}^N\left({\mathbf{\text{x}}}_i-\beta \mathbf{\text{m}}\right){\left({\mathbf{\text{x}}}_i-\beta \mathbf{\text{m}}\right)}^{+}\right]\mathbf{\text{h}}\\ & ={\mathbf{\text{h}}}^{+}{\mathbf{\text{C}}}_x^{\beta }\mathbf{\text{h}}\end{array}$$

(49)

where

$${\mathbf{\text{C}}}_x^{\beta }=\frac{1}{N}\sum _{i=1}^N\left({\mathbf{\text{x}}}_i-\beta \mathbf{\text{m}}\right){\left({\mathbf{\text{x}}}_i-\beta \mathbf{\text{m}}\right)}^{+}$$

(50)

Thus, AICH can be described as the average of the correlation peak intensities of N exemplars where the i-th exemplar (x_i − βm) is the i-th training image with part of the mean subtracted. Hence, it is desirable for all images in the training set to follow these exemplars' behavior. This can be done by forcing every image in the training set x_i to have a similar correlation output plane to an ideal correlation output shape f. To find the f that best matches all these exemplars' correlation output planes, its deviation from their correlation planes is minimized. This deviation can be quantified by ASE, defined as

$$\text{ASE}=\frac{1}{N}\sum _{i=1}^N{\left({\mathbf{\text{g}}}_i-\mathbf{\text{f}}\right)}^{+}\left({\mathbf{\text{g}}}_i-\mathbf{\text{f}}\right)$$

(51)

where,

$${\mathbf{\text{g}}}_i=({\mathbf{\text{X}}}_i-\beta \mathbf{\text{M}}){\mathbf{\text{h}}}^{\ast }$$

(52)

In Equation (52), the superscript * represents the complex conjugate operation. To find the optimum shape vector f_opt, the gradient of ASE with respect to f is set to zero, yielding

$$\begin{array}{cc}{\mathbf{\text{f}}}_{\text{opt}}& =\frac{1}{N}{\mathbf{\text{g}}}_i\\ & =\frac{1}{N}\sum _{i=1}^N({\mathbf{\text{X}}}_i-\beta \mathbf{\text{M}}){\mathbf{\text{h}}}^{\ast }\\ & =\left(1-\beta \right)\mathbf{\text{M}}{\mathbf{\text{h}}}^{\ast }\end{array}$$

(53)

The ASM is modified such that it measures the dissimilarity of the training images to (1 − β)Mh*. This new measure is called as the modified ASM (MASM), given by

$$\begin{array}{cc}\text{MASM}& =\frac{1}{N}\sum _{i=1}^N{\left[{\mathbf{\text{X}}}_i{\mathbf{\text{h}}}^{\ast }-(1-\beta )\mathbf{\text{M}}{\mathbf{\text{h}}}^{\ast }\right]}^{+}\left[{\mathbf{\text{X}}}_i{\mathbf{\text{h}}}^{\ast }-(1-\beta )\mathbf{\text{M}}{\mathbf{\text{h}}}^{\ast }\right]\\ & ={\mathbf{\text{h}}}^{\prime }\left\{\frac{1}{N}\sum _{i=1}^N{\left[{\mathbf{\text{X}}}_i-(1-\beta )\mathbf{\text{M}}\right]}^{\ast }\left[{\mathbf{\text{X}}}_i-(1-\beta )\mathbf{\text{M}}\right]\right\}{\mathbf{\text{h}}}^{\ast }\\ & ={\mathbf{\text{h}}}^{\prime }{\mathbf{\text{S}}}_x^{\beta }{\mathbf{\text{h}}}^{\ast }\\ & ={\mathbf{\text{h}}}^{+}{\mathbf{\text{S}}}_x^{\beta }\mathbf{\text{h}}\end{array}$$

(54)

where the superscript' represents the transpose operation and where it is considered that MASM is real in deriving the last equality in Equation (54). The diagonal matrix ${\mathbf{\text{S}}}_x^{\beta }$ is given by

$${\mathbf{\text{S}}}_x^{\beta }=\frac{1}{N}\sum _{i=1}^N{\left[{\mathbf{\text{X}}}_i-(1-\beta )\mathbf{\text{M}}\right]}^{\ast }\left[{\mathbf{\text{X}}}_i-(1-\beta )\mathbf{\text{M}}\right]$$

(55)

The ASM is a good measure for distortion tolerance; however, it lacks some discrimination capability that explains part of the MACH filter's inability to reject some clutter images. On the other hand, the MASM measure captures finer details of the training set that makes the EMACH filter more sensitive against clutter.

By maximizing the AICH and minimizing the MASM while controlling the parameter β, it is expected to explicitly keep a balance between the distortion tolerance and clutter rejection performance. Therefore, it is necessary to optimize the following new criterion

$$J^{\beta }(\mathbf{\text{h}})=\frac{\text{AICH}}{{\mathbf{\text{h}}}^{+}\gamma \mathbf{\text{Ih}}+{\mathbf{\text{h}}}^{+}{\mathbf{\text{S}}}_x^{\beta }\mathbf{\text{h}}}=\frac{{\mathbf{\text{h}}}^{+}{\mathbf{\text{C}}}_x^{\beta }\mathbf{\text{h}}}{{\mathbf{\text{h}}}^{+}(\gamma \mathbf{\text{I}}+{\mathbf{\text{S}}}_x^{\beta })\mathbf{\text{h}}}$$

(56)

where h⁺ γIh is the ONV term assuming an additive white noise with variance γ. The ONV helps to maintain noise tolerance when β increases, especially, at those low energy components. By maximizing the preceding criterion, the following condition is obtained for the EMACH filter

$${(\gamma \mathbf{\text{I}}+{\mathbf{\text{S}}}_x^{\beta })}^{-1}{\mathbf{\text{C}}}_x^{\beta }\mathbf{\text{h}}=\lambda \mathbf{\text{h}}$$

(57)

where λ is a scalar identical to J^β(h). Thus, h must be an eigenvector of ${(\gamma \mathbf{\text{I}}+{\mathbf{\text{S}}}_x^{\beta })}^{-1}{\mathbf{\text{C}}}_x^{\beta }$ with the corresponding eigenvalue λ. Since λ is identical to J^β(h), h should be the eigenvector that corresponds to the maximum eigenvalue. The other eigenvectors corresponding to the other nonzero eigenvalues provide smaller J^β(h) values. However, they may provide better discriminatory performance, as β is not known a priori. So the EMACH filter may be expressed as

$$\mathbf{\text{h}}=\text{Dominant eigenvector}\left\{{(\gamma \mathbf{\text{I}}+{\mathbf{\text{S}}}_x^{\beta })}^{-1}{\mathbf{\text{C}}}_x^{\beta }\right\}$$

(58)

9. Distance Classifier Correlation Filter (DCCF)

This is a correlation-based distance classifier scheme for recognition and classification of multiple similar or dissimilar objects. The underlying theory uses shift-invariant filters to compute distances between the input image and ideal references under an optimum transformation. The two ideas of relaxing the constraints on the correlation values at the origin and looking at the entire correlation plane rather than just the peak value led to the development of distance classifier correlation filters DCCFs [26–28]. The DCCF formulation can be used with any number of classes.

In the DCCF design, a global transformation is determined such that the transformed images from the same class are close to each other, whereas transformed images from different classes are separated from each other. This global transform leads to one correlation filter for each class. The use of these correlation filters is similar to the use of other correlation filters except in the final step. The test image is correlated with the correlation filter, and the resulting correlation peak is determined. This correlation-peak value is used to determine the distance of the test image to this class. Distances of the test image to all classes are determined and the class yielding the smallest distance is chosen. This paradigm allows for the relaxation of the correlation output constraints and the use of the entire correlation output.

It is important to realize that the use of DCCFs is similar to the use of other correlation filters. This means that the correlation peaks move by the same amount corresponding to the shift in the input, i.e., this is a shift-invariant operation. The DCCF concept has demonstrated promising performance on both infrared as well as synthetic aperture radar imagery [28]. Test results show that DCCFs outperform other correlation filters in recognizing targets while rejecting noise, clutter, and other confusing objects.

It is assumed that the training images are segmented and registered at a desired point. Fourier transform of an image x(m,n) of size ρ₁ × ρ₂ containing ρ = ρ₁ρ₂ pixels can be expressed as a ρ × 1 dimensional column vector x or as a ρ × ρ diagonal matrix X with the elements of x as its diagonal elements, i.e., diagonal {x} = X. Sometimes, the same quantity may be expressed both as a vector, say m_x, and as a diagonal matrix M_x. This implies that Hm_x and M_xh are equivalent.

The distance classifier uses a global transform denoted by H to separate the classes maximally while making them as compact as possible. For shift invariance, this transform matrix must be diagonal in the frequency domain. Multiplication of a vector x by a diagonal matrix H is equivalent to multiplying X(k,l) by H(k,l).

Here, a general C-class distance-classifier problem is analyzed by assuming that the peak correlation values are as different as possible for each of the classes although hard constraints are not used to enforce this. In addition, for each class, the correlation planes or their inverse Fourier transforms should be almost similar to the transformed ideal reference shape for this class. The correlation peaks are most likely at the origin for the registered training images but can occur elsewhere in the test cases depending on the location of the target.

Assume x_ik is the ρ-dimensional column vector containing the Fourier transform of the i-th image of the k-th class, 1 ≤ i ≤ N and 1 ≤ k ≤ C, where each class contains N training images. If m_k is the mean Fourier transform of class k, then

$$\begin{array}{cc}{\mathbf{\text{m}}}_k=\frac{1}{N}\sum _{i=1}^N{\mathbf{\text{x}}}_{ik},& 1\le k\le C\end{array}$$

(59)

The correlation peak at the origin between the mean training image from the k-th class and the filter h is given by ${\mathbf{\text{m}}}_k^{+}\mathbf{\text{h}}$. Thus, the overall mean Fourier transform of the entire training set becomes

$$\mathbf{\text{m}}=\frac{1}{C}\sum _{k=1}^C{\mathbf{\text{m}}}_k$$

(60)

The correlation peak of the overall mean image m with the filter h is given by

$${\mathbf{\text{m}}}^{+}\mathbf{\text{h}}=\frac{1}{C}\sum _{k=1}^C{\mathbf{\text{m}}}_k^{+}\mathbf{\text{h}}$$

(61)

which represents the overall average of the origin values of all correlation planes.

If the transformation h makes the in-class correlation planes similar, then the in-class peak values should be similar to each other and to their mean. Thus, to make the interclass separation between the correlation peaks large, the mean peak values of the classes are made as different as possible. Although several possible criteria might achieve this objective, the approach here is to increase the distance of all classes from the central mean. Toward this end, the following distance measure, called class separation, has been formulated

$$\begin{array}{cc}A(\mathbf{\text{h}})& =\frac{1}{C}\sum _{k=1}^C{\left|{\mathbf{\text{m}}}_k^{+}\mathbf{\text{h}}-{\mathbf{\text{m}}}^{+}\mathbf{\text{h}}\right|}^2\\ & =\frac{1}{C}\sum _{k=1}^C{\mathbf{\text{h}}}^{+}(\mathbf{\text{m}}-{\mathbf{\text{m}}}_k){(\mathbf{\text{m}}-{\mathbf{\text{m}}}_k)}^{+}\mathbf{\text{h}}\\ & ={\mathbf{\text{h}}}^{+}\mathbf{\text{Wh}}\end{array}$$

(62)

where

$$\mathbf{\text{W}}=\frac{1}{C}\sum _{k=1}^C(\mathbf{\text{m}}-{\mathbf{\text{m}}}_k){(\mathbf{\text{m}}-{\mathbf{\text{m}}}_k)}^{+}$$

(63)

is a ρ × ρ, full (i.e., non-diagonal) matrix of rank less than or equal to (C-1). The rank of W is less than or equal to (C-1), because it is obtained by the addition of C outer products of vectors (m − m_k), but these C vectors add up to a zero vector. If A(h) of Equation (62) is maximized, the class mean correlation peaks $({\mathbf{\text{m}}}_k^{+}\mathbf{\text{h}})$ will differ significantly. It is also desired that the distance of transformed inputs to their average be small. This distance B(h), which measures the compactness of each class, is the same as ASM defined for each class as

$$\begin{array}{cc}{\text{ASM}}_k=\frac{1}{N}\sum _{i=1}^N{\left|{\mathbf{\text{g}}}_{ik}-{\overline{\mathbf{\text{g}}}}_k\right|}^2& 1\le k\le C\end{array}$$

(64)

where

$$\begin{array}{c}{\mathbf{\text{g}}}_{ik}={\mathbf{\text{X}}}_{ik}{\mathbf{\text{h}}}^{\ast }\\ {\overline{\mathbf{\text{g}}}}_k={\mathbf{\text{M}}}_k{\mathbf{\text{h}}}^{\ast }\end{array}$$

(65)

are the Fourier transforms of the correlation outputs due to the i-th training image x_ik and the average training image m_k, respectively from class k. Note that X_ik and M_k are diagonal matrices with x_ik and m_k along the diagonal. The ASM is a measure of the similarity of the training images of a class to their mean and hence a measure of the compactness of the class after transformation by H. Using Equations (64) and (65), ASM_k can be rewritten as

$$\begin{array}{cc}{\text{ASM}}_k& =\frac{1}{N}\sum _{i=1}^N{({\mathbf{\text{X}}}_{ik}{\mathbf{\text{h}}}^{\ast }-{\mathbf{\text{M}}}_k{\mathbf{\text{h}}}^{\ast })}^{+}({\mathbf{\text{X}}}_{ik}{\mathbf{\text{h}}}^{\ast }-{\mathbf{\text{M}}}_k{\mathbf{\text{h}}}^{\ast })\\ & ={\mathbf{\text{h}}}^{\prime }\left[\frac{1}{N}\sum _{i=1}^N{({\mathbf{\text{X}}}_{ik}-{\mathbf{\text{M}}}_k)}^{\ast }({\mathbf{\text{X}}}_{ik}-{\mathbf{\text{M}}}_k)\right]{\mathbf{\text{h}}}^{\ast }\\ & ={\mathbf{\text{h}}}^{+}{\mathbf{\text{S}}}_k\mathbf{\text{h}}\end{array}$$

(66)

where

$${\mathbf{\text{S}}}_k=\frac{1}{N}\sum _{i=1}^N{({\mathbf{\text{X}}}_{ik}-{\mathbf{\text{M}}}_k)}^{\ast }({\mathbf{\text{X}}}_{ik}-{\mathbf{\text{M}}}_k)$$

(67)

In Equation (67), S_k is a ρ × ρ diagonal matrix where each training image contains ρ pixels. The overall ASM for C classes is defined as

$$\text{ASM}=B(\mathbf{\text{h}})=\frac{1}{N}\sum _{k=1}^C{\mathbf{\text{h}}}^{+}{\mathbf{\text{S}}}_k\mathbf{\text{h}}={\mathbf{\text{h}}}^{+}\mathbf{\text{Sh}}$$

(68)

where

$$\mathbf{\text{S}}=\frac{1}{C}\sum _{k=1}^C{\mathbf{\text{S}}}_k$$

(69)

To make the in-class metric B(h) small and to make the inter-class distance metric A(h) large, the filter h is designed to maximize the ratio

$$J(\mathbf{\text{h}})=\frac{A(\mathbf{\text{h}})}{B(\mathbf{\text{h}})}=\frac{{\mathbf{\text{h}}}^{+}\mathbf{\text{Wh}}}{{\mathbf{\text{h}}}^{+}\mathbf{\text{Sh}}}$$

(70)

with respect to h. The filter h that maximizes J(h) in Equation (70) is the eigenvector of S⁻¹W with the largest eigenvalue [28]. Because W is a non-diagonal matrix of rank less than or equal to (C-1), finding the dominant eigenvector of S⁻¹W requires a special algorithm when the training images, and thus the desired filter is of larger size [28]. When J(h) is maximum, the correlation shape produced by an input image is expected to be similar to the mean shape for its true class with a peak value different from the average peak value of any other class. The DCCF filter may be expressed as

$$\mathbf{\text{h}}=\text{Dominant eigenvector}\left\{{\mathbf{\text{S}}}^{-1}\mathbf{\text{W}}\right\}$$

(71)

The DCCF is the first technique proposed for shift-invariant transform-domain distance calculations with a correlator and that are specifically designed to accommodate multiple targets at different locations in the same image. This filter deals with the entire correlation plane and not just one point at the origin. It transforms the input image into a new space in which the distance of a test input from the classes is computed. Given a test input z, the distances d_k between the transformed input and the ideal shape for class k is computed as

$$\begin{array}{cc}{d}_k& ={\left|{\mathbf{\text{H}}}^{\ast }\mathbf{\text{z}}-{\mathbf{\text{H}}}^{\ast }{\mathbf{\text{m}}}_k\right|}^2\\ & ={\left|{\mathbf{\text{H}}}^{\ast }\mathbf{\text{z}}\right|}^2+{\left|{\mathbf{\text{H}}}^{\ast }{\mathbf{\text{m}}}_k\right|}^2-2\Re \left\{{\mathbf{\text{z}}}^{+}\mathbf{\text{H}}{\mathbf{\text{H}}}^{\ast }{\mathbf{\text{m}}}_k\right\}\\ & =p+b_k-2\Re \left\{{\mathbf{\text{z}}}^{+}{\mathbf{\text{h}}}_k\right\}\end{array}$$

(72)

In Equation (72), p = |H*z|² is the energy (independent of the class) of the transformed input test image z; b_k = |H*m_k|² is the energy (independent of the z) of the transformed mean of class k and h_k = HH*m_k is viewed as the effective filter for class k. Because there are only C classes for which distances must be computed, only C such filters are required.

In general, the targets may be anywhere in the input image. For shift-invariant distance calculation the interest is in the smallest value of d_k over all possible shifts of the target with respect to the class references. In Equation (72), because p and b_k are both positive and independent of the position of the target, the smallest value of d_k over all shifts is obtained when the third term (i.e., z⁺h_k) is as large as possible. Therefore, this term is chosen as the peak value for full cross correlation of z and h_k.

10. Polynomial DCCF (PDCCF)

Linear transformations such as the DCCF are attractive because of their optimality when the underlying statistics are Gaussian with equal covariances [29]. However, the DCCF uses transformations based on the second order statistics and does not capture higher-order statistics in images. Hence, it does not necessarily capture all of the discrimination information in some cases. Examples of such cases are encountered when signal dependent or multiplicative noises are present, and when inputs have non-Gaussian statistics. In non-Gaussian statistics cases, DCCF capabilities may be improved by applying different nonlinearities to the input image. By using nonlinear transformations, it may be possible to extract more useful information for discrimination. Thus, the classes that are not well separated in the original image space may become more separated in the nonlinearly mapped space. Also, point nonlinearities are preferred because they reduce the computational complexity because a simple nonlinearity is being applied to each point without considering all points in the neighborhood.

The polynomial DCCF (DCCF) extends the DCCF to include point nonlinear mappings of the input patterns [29]. Examples of such nonlinear mappings correspond to powers of pixels of the input images. Even though the resulting PDCCF system is not linear with respect to the input patterns, it is still linear in the kernel. This property allows frequency domain techniques for the design, analysis, and implementation of this filter. Another important property is that this system works on different powers of input image pixels, which corresponds to a multi-dimensional correlation operation and thus extends the linear DCCF classification optimization criterion to a nonlinear one, and no nonlinear optimization is involved. Moreover, the PDCCF system provides a new framework for combining different correlation filters, where each filter in the system is optimized jointly with other filters.

The PDCCF first maps the input image x(m,n) into x^j(m,n) via point nonlinearities η_j, where j = 1, 2, …, n. Thus x^j(m,n) is related to x(m,n) through the following relationship

$${\eta }_j:x(m,n)\to x^j(m,n)$$

(73)

In Equation (73), all x^j(m,n) are assumed to have the same size as the input x(m,n). Examples of such mappings include various powers, logarithms, cosines, etc. The nonlinearly mapped input image x(m,n) is transformed by the filter h_j(m,n) built using x^j(m,n). The overall distance is obtained by adding the distances resulting from the shift-invariant minimum mean squared error computations between every transformed image of the input and its respective ideal transformed reference. Those ideal transformed references are computed a priori by using the nonlinear functions η₁, η₂, …, and η_n followed by the application of the filters h₁(m,n), h₂(m,n), …, and h_n(m,n), respectively.

In the linear DCCF, the filtered image g(m,n) resulting from transformation of x¹(m,n) by h₁(m,n) can be written as

$$g(m,n)=h_1(m,n)\odot x^1(m,n)$$

(74)

where, ⊙ denotes the crosscorrelation and x¹(m,n) = x(m,n). By augmenting the input image with x²(m,n), another term can be added involving the crosscorrelation of the x²(m,n) with h₂(m,n) to obtain an output transformed by the two-term PDCCF, as shown below.

$$g(m,n)=h_1(m,n)\odot x^1(m,n)+h_2(m,n)\odot x^2(m,n)$$

(75)

Thus, the PDCCF has more terms at its disposal with which it can achieve better discrimination. Clearly, these new nonlinear versions of inputs are completely dependent on the original inputs, and in that sense no new information is being created. However, the new representations enable the correlation filters to provide better recognition. By continuing to add more terms such as h_j(m,n)⊙x^j(m,n), the n-term PDCCF can be obtained as

$$g(m,n)=\sum _{j=1}^n{h}_j(m,n)\odot x^j(m,n)$$

(76)

If the focus is on the point-wise power nonlinearities for all η_j's the nonlinear mapping of the input, x^j(m,n) can be defined as

$$x^j(m,n)={[x(m,n)]}^j$$

(77)

where, j ∈ ℝ∞. The power nonlinearity plays an important role in SAR images as it can enhance the bright scatters or the overall contrast [29]. The filters h_j(m,n) are computed jointly, and thus the advantages of the closed form solutions and nonlinear systems are combined and exploited. In the following analysis, Ψ represents the set of all powers used to construct a particular PDCCF. Although the nonlinearity is applied in the spatial domain, in the analyses to follow (e.g., filter formation, distance calculation, etc.) all the quantities are actually in Fourier domain. For example, x is a vector obtained by lexicographical rearrangement of the Fourier transform of x(m,n). All vectors and matrices with superscript j (e.g., x^j and X^j) represent these vectors and matrices (x and X) with each of their elements in the image (spatial) domain raised to the j-th power.

Assume ${\mathbf{\text{m}}}_k^j$ is the mean of class k of the Fourier transforms of training images resulting from raising all their pixels to the j-th power and h_j is the filter built for images raised to the j-th power. Each of the Fourier transforms of the original image along with the Fourier transforms of its variations can be represented by use of a single block vector. Thus, the Fourier transform of the mean images of class k after augmentation becomes

$${\mathbf{\text{m}}}_k=\left(\begin{array}{l}{\mathbf{\text{m}}}_k^1\\ {\mathbf{\text{m}}}_k^2\\ .\\ .\\ {\mathbf{\text{m}}}_k^n\end{array}\right)$$

(78)

Further, the filters, h₁, h₂, … and h_n can be combined into one filter, h as follows

$$\mathbf{\text{h}}=\left(\begin{array}{l}{\mathbf{\text{h}}}_1\\ {\mathbf{\text{h}}}_2\\ .\\ .\\ {\mathbf{\text{h}}}_n\end{array}\right)$$

(79)

With Equations (75), (78), and (79), the correlation peak at the origin produced in response to the mean image of class k is given by

$${\overline{g}}_k(0,0)=\sum _{j=1}^n{\mathbf{\text{h}}}_j^{+}{\mathbf{\text{m}}}_k^j={\mathbf{\text{h}}}^{+}{\mathbf{\text{m}}}_k$$

(80)

The distance between the classes, after being augmented and then transformed by the filters h₁, h₂, …, and h_n can be expressed as

$$\begin{array}{cc}A(\mathbf{\text{h}})& =\frac{1}{C}\sum _{k=1}^C{\left|\sum _{j=1}^n{\mathbf{\text{h}}}_j^{+}{\mathbf{\text{m}}}_k^j-\sum _{j=1}^n{\mathbf{\text{h}}}_j^{+}{\mathbf{\text{m}}}^j\right|}^2\\ & =\frac{1}{C}\sum _{k=1}^C{\left|{\mathbf{\text{h}}}^{+}{\mathbf{\text{m}}}_k-{\mathbf{\text{h}}}^{+}\mathbf{\text{m}}\right|}^2\\ & ={\mathbf{\text{h}}}^{+}\mathbf{\text{Wh}}\end{array}$$

(81)

where

$$\mathbf{\text{W}}=\frac{1}{C}\sum _{k=1}^C({\mathbf{\text{m}}}_k-\mathbf{\text{m}}){({\mathbf{\text{m}}}_k-\mathbf{\text{m}})}^{+}$$

(82)

To separate the classes as much as possible, the filter is required to produce a large A(h). Simultaneously, the compactness of the classes needs to be increased after transformation by h₁, h₂, …, and h_n. The compactness is measured by the similarity of the training images of a class to their mean. It can be represented by the ASM of that class. In general, the ASM for class k is defined as

$$\begin{array}{cc}{\text{ASM}}_k=\frac{1}{N}\sum _{i=1}^N{\left|{\mathbf{\text{g}}}_{ik}-{\overline{\mathbf{\text{g}}}}_k\right|}^2& 1\le k\le C\end{array}$$

(83)

where

$$\begin{array}{c}{\mathbf{\text{g}}}_{ik}={\mathbf{\text{X}}}_{ik}^j{\mathbf{\text{h}}}_j^{\ast }\\ {\overline{\mathbf{\text{g}}}}_k={\mathbf{\text{M}}}_k^j{\mathbf{\text{h}}}_j^{\ast }\end{array}$$

(81)

are the Fourier transforms of the filtered images produced by the transform filters in response to the input image x^j and the mean image m_k, respectively. Thus, from Equations (83) and (84), ASM_k can be written as

$$\begin{array}{cc}{\text{ASM}}_k& =\frac{1}{N}\sum _{i=1}^N{\left|{\mathbf{\text{X}}}_{ik}^1{\mathbf{\text{h}}}_1^{\ast }+{\mathbf{\text{X}}}_{ik}^2{\mathbf{\text{h}}}_2^{\ast }+\dots +{\mathbf{\text{X}}}_{ik}^n{\mathbf{\text{h}}}_n^{\ast }-{\mathbf{\text{M}}}_k^1{\mathbf{\text{h}}}_1^{\ast }-{\mathbf{\text{M}}}_k^2{\mathbf{\text{h}}}_2^{\ast }-\mathrm{...}-{\mathbf{\text{M}}}_k^2{\mathbf{\text{h}}}_n^{\ast }\right|}^2\\ & =\frac{1}{N}\sum _{i=1}^N{\left[({\mathbf{\text{X}}}_{ik}^1-{\mathbf{\text{M}}}_k^1){\mathbf{\text{h}}}_1^{\ast }+({\mathbf{\text{X}}}_{ik}^2-{\mathbf{\text{M}}}_k^2){\mathbf{\text{h}}}_2^{\ast }+\dots +({\mathbf{\text{X}}}_{ik}^n-{\mathbf{\text{M}}}_k^n){\mathbf{\text{h}}}_n^{\ast }\right]}^{+}\left[({\mathbf{\text{X}}}_{ik}^1-{\mathbf{\text{M}}}_k^1){\mathbf{\text{h}}}_1^{\ast }+({\mathbf{\text{X}}}_{ik}^2-{\mathbf{\text{M}}}_k^2){\mathbf{\text{h}}}_2^{\ast }+\dots +({\mathbf{\text{X}}}_{ik}^n-{\mathbf{\text{M}}}_k^n){\mathbf{\text{h}}}_n^{\ast }\right]\\ & =\sum _{u=1}^n\sum _{v=1}^n{\mathbf{\text{h}}}_u^{+}{\mathbf{\text{S}}}_{\mathit{\text{kuv}}}{\mathbf{\text{h}}}_v\\ & =({\mathbf{\text{h}}}_1^{+}{\mathbf{\text{h}}}_2^{+}\mathrm{...}{\mathbf{\text{h}}}_n^{+})\left[\begin{array}{llll}{\mathbf{\text{S}}}_{k11}\hfill & {\mathbf{\text{S}}}_{k12}\hfill & \dots \hfill & {\mathbf{\text{S}}}_{k1n}\hfill \\ {\mathbf{\text{S}}}_{k21}\hfill & {\mathbf{\text{S}}}_{k22}\hfill & \dots \hfill & {\mathbf{\text{S}}}_{k2n}\hfill \\ .\hfill & \hfill & \hfill & \hfill \\ .\hfill & \hfill & \hfill & \hfill \\ {\mathbf{\text{S}}}_{kn1}\hfill & {\mathbf{\text{S}}}_{kn2}\hfill & \dots \hfill & {\mathbf{\text{S}}}_{\mathit{\text{knn}}}\hfill \end{array}\right]\left(\begin{array}{l}{\mathbf{\text{h}}}_1\\ {\mathbf{\text{h}}}_2\\ .\\ .\\ {\mathbf{\text{h}}}_n\end{array}\right)\\ & ={\mathbf{\text{h}}}^{+}{\mathbf{\text{S}}}_k\mathbf{\text{h}}\end{array}$$

(85)

where

$${\mathbf{\text{S}}}_{\mathit{\text{kuv}}}=\left[\frac{1}{N}\sum _{i=1}^N{\mathbf{\text{X}}}_{ik}^u{({\mathbf{\text{X}}}_{ik}^v)}^{\ast }\right]-{\mathbf{\text{M}}}_k^u{({\mathbf{\text{M}}}_k^v)}^{\ast }$$

(86)

are all diagonal matrices and

$${\mathbf{\text{S}}}_k=\left[\begin{array}{llll}{\mathbf{\text{S}}}_{k11}\hfill & {\mathbf{\text{S}}}_{k12}\hfill & \dots \hfill & {\mathbf{\text{S}}}_{k1n}\hfill \\ {\mathbf{\text{S}}}_{k21}\hfill & {\mathbf{\text{S}}}_{k22}\hfill & \dots \hfill & {\mathbf{\text{S}}}_{k2n}\hfill \\ .\hfill & \hfill & \hfill & \hfill \\ .\hfill & \hfill & \hfill & \hfill \\ {\mathbf{\text{S}}}_{kn1}\hfill & {\mathbf{\text{S}}}_{kn2}\hfill & \dots \hfill & {\mathbf{\text{S}}}_{\mathit{\text{knn}}}\hfill \end{array}\right]$$

(87)

The overall ASM for C classes is then defined as

$$\text{ASM}=B(\mathbf{\text{h}})=\frac{1}{N}\sum _{k=1}^C{\mathbf{\text{h}}}^{+}{\mathbf{\text{S}}}_k\mathbf{\text{h}}={\mathbf{\text{h}}}^{+}\mathbf{\text{Sh}}$$

(88)

where

$$\mathbf{\text{S}}=\frac{1}{C}\sum _{k=1}^C{\mathbf{\text{S}}}_k$$

(89)

The filter h that maximizes the ratio A(h)/B(h), is the dominant eigenvector of S⁻¹W as in case of DCCF and given by

$$\mathbf{\text{h}}=\text{Dominant eigenvector}\left\{{\mathbf{\text{S}}}^{-1}\mathbf{\text{W}}\right\}$$

(90)

Given a test input z, the distances d_k between the transformed input and the ideal shape for class k is computed by using MSE-then-total approach [30]. In this approach, n distances are computed for class k. The j-th distance, d_jk, is defined as follows

$$\begin{array}{cc}{d}_{jk}={\left|{\mathbf{\text{H}}}_j^{\ast }{\mathbf{\text{z}}}^j-{\mathbf{\text{H}}}_j^{\ast }{\mathbf{\text{m}}}_k^j\right|}^2& 1\le j\le n\end{array}$$

(91)

The distance d_jk can be rewritten as

$$d_{jk}=p_j+b_{jk}-2\Re \left\{{({\mathbf{\text{z}}}^j)}^{+}{\mathbf{\text{h}}}_{jk}\right\}$$

(92)

where

$$p_j={\left|{\mathbf{\text{H}}}_j^{\ast }{\mathbf{\text{z}}}^j\right|}^2$$

(93)

$$b_{jk}={\left|{\mathbf{\text{H}}}_j^{\ast }{\mathbf{\text{m}}}_k^j\right|}^2$$

(94)

$${\mathbf{\text{h}}}_{jk}={\mathbf{\text{H}}}_j{\mathbf{\text{H}}}_j^{\ast }{\mathbf{\text{m}}}_k^j$$

(95)

The inner products, shown as the third terms in Equation (92), are between the j-th variation of the input, z^j and the corresponding j-th filter, h_kj. The total distance d_k to a class is then found by

$$d_k=\sum _{j=1}^n{d}_{jk}$$

(96)

The input image is assigned to the class with the least total distance.

11. Target Tracking in FLIR Imagery

In general, tracking of a moving pattern/target requires recognizing and then locating the target in a scene, finding the target motion, understanding the direction of motion of the target, and then following that target as it moves through the sequence of image frames. The detection and tracking of desired targets in a real life image corrupted by noise, clutter, illumination and other three-dimensional (3D) artifacts, poses a very complex problem and demands sophisticated solutions using pattern recognition and motion estimation methods [31,32]. Things become more complicated if there are more than one target in the scene and simultaneous multiple targets tracking is required.

Forward-looking infrared (FLIR) images are frequently used in automatic target recognition (ATR) applications. It is challenging to detect and track targets in FLIR imagery. To detect independent moving objects in FLIR image sequences, the sensor properties have to be taken into account. Additional challenges are caused due to many important differences of FLIR images with visual sequences [33,34]. Many researchers have investigated various approaches for detection, recognition, classification and pose estimation of targets from FLIR images including both matched spatial filter (MSF) based correlators and joint transform correlators [33–40]. However, the application of MSFs or their variants (e.g., MACH, DCCF) for the FLIR imagery is very limited; although those have been used for the simulated and real synthetic aperture radar (SAR) and laser radar (LADAR) imagery [24,41–47].

In this section, three different algorithms are demonstrated for pattern recognition and tracking based on the combination of the detection and classification filters [48–51]:

• MACH filter-based detection and DCCF-based classification (MACH-DCCF)

• MACH filter-based detection and PDCCF-based classification (MACH-PDCCF)

• EMACH filter-based detection and PDCCF-based classification (EMACH-PDCCF)

The detection filters are trained by the target images of expected size and orientation variations with expected size of input scene. The classification filters are formulated with the expected size of target images and trained by the target images of expected size and orientation variations. The first step of the real time system is detection, which involves correlating the input scene with all detection filters (one for each desired or expected target class) and combining the correlation outputs. In the second step, a predefined number of ROIs having the expected size of target images are selected based on the regions having higher correlation peak values in the combined correlation output. To ensure that all desired or expected targets are included in the ROIs, the number of ROIs should be at least three times higher than the number of expected targets. Classification filters are then applied to these ROIs and target types along with clutters are identified based on a distance measure and a threshold. Moving target detection and tracking are accomplished by following this technique for all incoming image frames by applying the same filters.

Multiple detection filters and classification filters are formulated for each target based on different size ranges or aspect angles. All of the filters for different ranges can be applied simultaneously through the whole range of the image sequence and decision can be made based on the output of the filter corresponding to the highest correlation peak or minimum distance. However, in this illustration, one detection filter and one classification filter is used for each class of targets for a particular range. A block diagram of the method for real time pattern recognition and tracking is shown in Figure 1 for a two-class detection system.

11.1. Image Dataset

The FLIR image database used in this research is supplied by Army Missile Command (AMCOM). This image database has a total of 50 real life infrared video sequences, some of which contain a single target in the scene and some contain multiple targets. In general, the image sequences are closing sequences, i.e., the targets become closer to the observer as the later frames appear in the scene. Thus, the size and signature of the targets change from the first frame to the last frame. Moreover, as the targets move, there are changes in the targets' orientations from one frame to the next. The database is also associated with ground truth data files containing the list of targets at each frame of each image sequence and their size and location in the frame. Among the 50 sequences, the proposed techniques have been applied for several single and multiple targets image sequences and the techniques are still being tested on other remaining sequences. For this paper, the analysis on 4 single target sequences (L1415, L2018, L2312, M1406), 2 two-target sequences (L1701, L1911) and 1 three-target sequence (L1618) have been reported.

11.2. Single-Target Image Sequences

At first, consider Sequence L2018, which has the highest difficulty level among the selected single target image sequences. In this sequence, there are a total of 448 frames, each of which contains the same target (tank1). The size of this target in the first frame is 3 × 4 pixels whereas that in the final frame is 12 × 23 pixels. For effective detection and classification, proper selection of the training images is important. It is obvious that a single detection or classification filter obtained by exploiting the training images from this large variation of sizes may lose its selectivity. So, for this particular sequence, two different detection filters (MACH or EMACH) are formed for the target to use in two ranges of the image frames depending on the size of the target.

The first range detection filters are trained using the target patches taken from the first 200 frames at an interval of 5 frames (the 1st, 5th, 10th, 15th, …, 200th frames). The ground truth data files are used to read the coordinates and sizes of the targets and then to segment out the target patches from the original frames. Each of these patches is then mean-subtracted and normalized dividing by the RMS value of the mean-subtracted patch. Thereafter, each patch is placed at the center of a 118 × 118-pixel zero padded matrix to form a full size training image for the detection filters. A sample training image of this type for Sequence L2018 is shown in Figure 2. The spatial domain representations of the MACH filter and EMACH filter for Range-1, trained by the above mentioned training images, are shown in Figure 3a,b, respectively.

The detection filters for Range-2, to use from Frame 201 to Frame 448, are trained by the target patches taken from the Frames 201 to 300 at an interval of 5 frames. The corresponding Range-2 MACH and EMACH filters are shown in Figure 4a,b, respectively.

In general, the classification filters (DCCF or PDCCF) are also required to be formulated for different ranges of the targets for better selectivity. The sizes of these filters are usually chosen as the expected size of the targets in the corresponding range of image frames. Target patches are taken as before but they are not normalized for classification filters. Each patch is placed at the center of a zero padded background having the size of the classification filter. If the patch is larger than the filter size, it is truncated at the sides. For Sequence L2018, it is found that the classification filter (either DCCF or PDCCF) of 12 × 16 pixels trained by the target patches of Frame 1 to Frame 200 at an interval of 4 frames works well for almost all frames. A 12 × 16-pixel sample training image of tank1 of Sequence L2018 for classification filter is shown in Figure 5.

It is assumed that the single target sequences are known to have only one target of interest. Hence, in this work, single class (1-class) classification filters (DCCF, PDCCF) are formulated for each particular sequence using the corresponding target patches. Although the classification filters actually need to be formulated using at least two classes of training images, in this work, 1-class filters are formulated by using slight modification in the basic formula. However, these 1-class approximated filters have been found to work well in most cases.

The training data or parameters for all the four filters (MACH, EMACH, DCCF, PDCCF) used for the three algorithms for detecting and tracking the target (tank1) in Sequence L2018 are summarized in

Table 1. Training data for Sequence L2018.

Target (No. of Frames)	Filter	Working Frame Range	Training Frames		Filter Size	β	γ	Ψ
			Range Taken	Interval Taken
tank1 (448)	MACH Range-1	1–200	1–200	5	118 × 118	-	0.1	-
	MACH Range-2	201–448	201–300	5	118 × 118	-	1.0	-
	EMACH Range-1	1–200	1–200	5	118 × 118	0.2	0.1	-
	EMACH Range-2	201–448	201–300	5	118 × 118	0.1	1.0	-
-	DCCF	1–448	1–200	4	12 × 16	-	-	-
-	PDCCF	1–448	1–200	4	12 × 16	-	-	1.0, 1.5, 2.0

. The first column of the table represents the name of the target along with the number of frames for which the ground truth data is available for it. The second column shows the filter names with their range indices; and the third column shows the range of the actual image frames where a particular filter is applied. The fourth and fifth columns provide the considered range of the frames and the order or interval of the frames used to take the target patch for training. In Table 1, γ is the filter parameter for MACH; β and γ are the filter parameters for EMACH and Ψ is the set of nonlinear powers used for PDCCF formulation.

Filter formulations of the three other single target sequences (L1415, L2312 and M1406) are also similar to Sequence L2018. To overcome the size variation from the initial frames to the final frames, multiple filters of multiple range bins are required in some cases to apply at different ranges. The training data for these three single target sequences are depicted in

Table 2. Training data for Sequence L1415.

Target (No. of Frames)	Filter	Working Frame Range	Training frames		Filter Size	β	γ	Ψ
			Range Taken	Interval Taken
Mantruck (281)	MACH	1–281	1–100	5	118 × 118	-	1	-
	EMACH	1–281	1–100	5	118 × 118	0.1	1	-
-	DCCF	1–281	1–100	5	12 × 16	-	-	-
-	PDCCF	1–281	1–100	5	6 × 8	-	-	1.0, 1.5, 2.0

–

Table 4. Training data for Sequence M1406.

Target (No. of Frames)	Filter	Working Frame Range	Training Frames		Filter Size	β	γ	Ψ
			Range Taken	Interval Taken
Bradley (380)	MACH	1–380	1–100	5	118 × 118	-	1	-
	EMACH	1–380	1–100	5	118 × 118	0.1	1	-
-	DCCF	1–380	1–300	10	14 × 16	-	-	-
-	PDCCF Range-1	1–200	1–100	5	8 × 10	-	-	1.0, 1.5, 2.0
-	PDCCF Range-2	201–380	201–300	5	8 × 10	-	-	1.0, 1.5, 2.0

, respectively.

For analysis, consider the first frame (Frame 1) of Sequence L2018 shown in Figure 6a, apply Fourier transform to it and then correlate with the Fourier domain filter MACH Range-1 to obtain the correlation output of Figure 6b. It is obvious that the correlation output contains false peaks and high-energy diffractions at low frequency which make the actual correlation signal negligible. For this reason, it requires 33 ROIs to include the target of interest as shown in Figure 7b.

To eliminate the strong low frequency components, a notch filter is used before applying the inverse Fourier transform operation, which actually suppresses the Fourier domain components along the axes to zero. The correlation output after applying this notch filter is shown in Figure 8a. Five ROIs in Frame 1, selected based on this correlation output, are shown in Figure 8b where the target (tank1) is included within these ROIs. This notch filter is used in the detection stage of each frame using either MACH or EMACH filter for all image sequences analyzed in this work. The detection results using EMACH filter are similar to MACH filter with or without the use of notch filter.

It is obvious that the detection and classification cannot be done correctly with detection filters alone. Because of the presence of multiple identical and different targets and clutters, the highest correlation peak or PSR value is not always guaranteed for the desired target. Therefore, classification filters are used for improved discrimination for the single target image sequences too. It is found that the 1-class classification filters work well in rejecting the clutters and backgrounds. However, to make a decision for identifying the target and rejecting the clutter the ROI having the least distance is considered as the potential target. If the distance to the ROI having the second minimum distance is not higher than the prescribed percentage of the least distance, then the ROI having the least distance is rejected as clutter or background. Since EMACH filter has improved clutter rejection capability, the application of EMACH filter in the detection stage facilitates in lowering the number of ROIs introduced into the classification stage. For this particular sequence (L2018), it is found in the simulation that 8 ROIs are required to include the target in almost all frames in the case of MACH filter, whereas 6 ROIs are sufficient with EMACH filter. However, for generality, 8 ROIs are also considered in case of EMACH filter for this sequence.

All single target sequences are tested by using the three developed algorithms: MACH-DCCF, MACH-PDCCF and EMACH-PDCCF. Some of the frames of Sequence L2018 showing the results after applying the EMACH-PDCCF algorithm are given in Figure 9. The tracking algorithm inserts a “T” mark at the locations of the detected targets (tank1) at each frame as shown in Figure 9. The number at the lower left corner of each frame indicates the frame number. In Figure 9, some sample frames are also included where the classification is incorrect or no decision is inferred. The detection and tracking results for all single target sequences are summarized for all the three algorithms in

Table 5. Tracking results of single-target sequences.

Seq. Name	Total Frames	No. of ROIs Taken	Threshold	Target Name	No. of Frames Target Present	No. of Frames Detected Correctly	Total No. of False Alarms	Percentage of Successful Detection
MACH-DCCF Algorithm
L1415	281	4	1.00	mantruck	281	281	0	100
L2018	448	8	0.99	tank1	448	357	36	80
L2312	368	4	1.00	APC1	368	366	2	99
M1406	380	4	1.00	Bradley	380	380	0	100
MACH-PDCCF Algorithm
L1415	281	4	1.00	mantruck	281	263	9	94
L2018	448	8	0.99	tank1	448	371	37	83
L2312	368	4	1.00	APC1	368	368	0	100
M1406	380	4	1.00	Bradley	380	353	27	93
EMACH-PDCCF Algorithm
L1415	281	4	1.00	mantruck	281	274	1	98
L2018	448	8	1.00	tank1	448	386	60	86
L2312	368	4	1.00	APC1	368	368	0	100
M1406	380	4	1.00	Bradley	380	354	26	93

. In the threshold column of the table, a single value indicates that a single threshold is used for all ranges. From Table 5, it is observed that the poorest performance is achieved for Sequence L2018 among the four single target sequences presented. This may be attributed to the scene complexities and the drawback of using same classification filters through all the ranges of the sequence.

11.3. Two-Target Image Sequences

To assess the performance of the algorithms with two targets present in the scene, consider Sequences L1701 and L1911. The two targets in Sequence L1701 are Bradley and pickup. Out of 388 frames of this sequence, ground truth data for Bradley is available for 371 frames and that for pickup is available for 43 frames. The pickup disappears from the scene at Frame 31 and reappears at Frame 81 and again disappears at a later frame. The two targets, APC1 and tank1, in Sequence L1911 are present in all the 165 frames of the sequence. In these image sequences, the target sizes increase significantly from the first frame to the final frame. Therefore, different filters must be used for different ranges. For a particular range, two detection filters (MACH or EMACH) and a 2-class classification filter (DCCF or PDCCF) are required for a two-target sequence.

Table 6. Training data for Sequence L1701.

Target (No. of Frames)	Filter	Working Frame Range	Training Frames		Filter Size	β	γ	Ψ
			Range Taken	Interval Taken
Bradley (371)	MACH Range-1	1–100	1–100	5	118 × 118	-	1.0	-
	MACH Range-2	101–200	101–200	5	118 × 118	-	1.0	-
	MACH Range-3	201–300	201–300	5	118 × 118	-	0.1	-
	MACH Range-4	301–388	301–370	5	118 × 118		0.1	-
	EMACH Range-1	1–100	1–100	5	118 × 118	0.1	1.0	-
	EMACH Range-2	101–200	101–200	5	118 × 118	0.1	1.0	-
	EMACH Range-3	201–300	201–300	5	118 × 118	0.1	0.1	-
	EMACH Range-4	301–388	301–370	5	118 × 118	0.1	0.1	-
Pickup (43)	MACH Range-1	1–388	1–43	3	118 × 118		1.0	-
	EMACH Range-1	1–388	1–43	3	118 × 118	0.1	1.0	-
-	DCCF Range-1	1–100	1–100	5	6 × 8	-	-	-
-	DCCF Range-2	101–200	201–300	5	8 × 12	-	-	-
-	DCCF Range-3	201–300	201–300	5	10 × 18	-	-	-
-	DCCF Range-4	301–388	301–370	5	10 × 18	-	-	-
-	PDCCF Range-1	1–100	1–100	5	8 × 8	-	-	1.0, 1.5, 2.0
-	PDCCF Range-2	101–200	101–200	5	8 × 12	-	-	1.0, 1.5, 2.0
-	PDCCF Range-3	201–300	201–300	5	10 × 18	-	-	1.0, 1.5, 2.0
-	PDCCF Range-4	301–388	301–370	5	10 × 18	-	-	1.0, 1.5, 2.0

shows the training data/parameters selected for different filters for Sequence L1701 while

Table 7. Training data for Sequence L1911.

Target (No. of Frames)	Filter	Working Frame Range	Training Frames		Filter Size	β	γ	Ψ
			Range Taken	Interval Taken
APC1 (165)	MACH Range-1	1:100	1:100	5	118 × 118	-	0.1	-
	MACH Range-2	101:165	101:150	5	118 × 118	-	0.1	-
	EMACH Range-1	1:100	1:100	5	118 × 118	0.1	0.1	-
	EMACH Range-2	101:165	101:150	5	118 × 118	0.1	0.1	-
tank1 (165)	MACH Range-1	1:100	1:100	5	118 × 118	-	0.1	-
	MACH Range-2	1:165	101:150	5	118 × 118	-	0.1	-
	EMACH Range-1	1:100	1:100	5	118 × 118	0.1	0.1	-
	EMACH Range-2	1:165	101:150	5	118 × 118	0.1	0.1	-
-	DCCF Range-1	1:100	1:100	5	8 × 16	-	-	-
-	DCCF Range-2	101:130	101:130	5	8 × 16	-	-	-
-	DCCF Range-3	131:165	131:160	5	16 × 36	-	-	-
-	PDCCF Range-1	1:130	1:100	5	8 × 16	-	-	1.0, 1.5, 2.0
-	PDCCF Range-2	1:130	1:100	5	8 × 16	-	-	1.0, 1.5, 2.0
-	PDCCF Range-3	131:165	131:160	5	16 × 36	-	-	1.0, 1.5, 2.0

displays the same parameters for Sequence L1911. It may be mentioned that even if one target disappears after a few frames, both the detection filters and the 2-class classification filter are continued to apply to the remaining image frames; because in reality, it is not known whether any target is going out or coming back. This ensures the detection of a target that may reappear after a few frames.

Using the detection and classification filters having the design parameters in Tables 6 and 7, all algorithms are tested for detection, classification and tracking of the objects in Sequences L1701 and L1911. The results obtained after applying the EMACH-PDCCF algorithm on Sequence L1701 are shown for some sample frames in Figure 10. The tracking algorithm inserts “T1” for the detected Class-1 type targets (Bradley) and “TII” for the detected Class-2 type targets (pickup) at their corresponding location in the image frame. The frame number is shown at the lower left corner of each frame displayed. From Figure 10, it is obvious that the tracking algorithm can successfully detect and classify the targets when they are present in the input scene. The complete tracking results for both the sequences are summarized in

Table 8. Tracking results of two-target sequences.

Seq. Name	Total Frames	No. of ROIs Taken	Threshold	Target Name	No. of Frames Target Present	No. of Frames Detected Correctly	Total No. of False Alarms	Percentage of Successful Detection
MACH-DCCF Algorithm
L1701	388	6	0.70, 0.30, 0.05, 0.04	Bradley	388	230	248	59
				pickup	45	26	2	58
L1911	165	6	0.40	APC1	-	-	-	Fails
				Tank1	-	-	-	Fails
MACH-PDCCF Algorithm
L1701	388	6	0.40, 0.05, 0.06, 0.06	Bradley	388	369	12	95
				pickup	45	28	1	62
L1911	165	6	0.40	APC1	165	152	0	92
				tank1	165	165	3	100
EMACH-PDCCF Algorithm
L1701	388	4	0.40, 0.05, 0.06, 0.06	Bradley	388	369	9	95
				pickup	45	31	8	69
L1911	165	4	0.40	APC1	165	160	0	97
				tank1	165	165	5	100

for all algorithms. In the threshold column of the table, a single value indicates that a single threshold is used for all ranges. Otherwise, different threshold values in that column displays the thresholds used for various ranges of classification filters employed. It is observed that MACH-DCCF algorithm fails for Sequence L1911 in detecting and classifying the targets, and rejecting the clutters. On the other hand, EMACH-PDCCF algorithm provides the best results considering all the factors simultaneously, such as, required number of ROIs, percentage of successful detection and the total number of false alarms.

11.4. Three-Target Image Sequences

The three target images in Sequence L1618 are APC1, M60 and truck. Out of 300 frames in this sequence, ground truth data for APC1 is available for 291 frames, ground truth data for M60 is found for 101 frames and that for truck is found for 6 frames. The truck disappears from the sequence at Frame 18 and M60 disappears from the sequence at Frame 104. Like the other sequences, the size of the targets increases significantly from the first frame to the final frame. Thus, different filters are required for different ranges of this sequence. Assuming three expected targets in the scene for any particular range, three detection filters (MACH or EMACH) and a three-class classification filter (DCCF or PDCCF) are required for every frame in each range of this sequence. The training data/parameters for different filters, chosen for Sequence L1618 are displayed in

Table 9. Training data for Sequence L1618.

Target (No. of Frames)	Filter	Working Frame Range	Training Frames		Filter Size	β	γ	Ψ
			Range Taken	Interval Taken
APC1 (291)	MACH Range-1	1–100	1–100	5	118 × 118	-	1.0	-
	MACH Range-2	101–200	101–200	5	118 × 118	-	1.0	-
	MACH Range-3	201–300	201–290	5	118 × 118	-	1.0	-
	EMACH Range-1	1–100	1–100	5	118 × 118	0.1	1.0	-
	EMACH Range-2	101–200	101–200	5	118 × 118	0.1	1.0	-
	EMACH Range-3	201–300	201–290	5	118 × 118	0.1	1.0	-
M60 (101)	MACH Range-1	1–300	1–100	5	118 × 118	-	1.0	-
	EMACH Range-1	1–300	1–100	5	118 × 118	0.1	1.0	-
Truck (6)	MACH Range-1	1–300	1–6	1	118 × 118	-	1.0	-
	EMACH Range-1	1–388	1–6	1	118 × 118	0.1	1.0
-	DCCF Range-1	1–100	1–100	5	6 × 12	-	-	-
-	DCCF Range-2	101–200	201–300	5	8 × 16	-	-	-
-	DCCF Range-3	201–300	201–290	5	8 × 16	-	-	-
-	PDCCF Range-1	1–100	1–100	5	6 × 12	-	-	1.0, 1.5, 2.0
-	PDCCF Range-2	101–200	101–200	5	8 × 16	-	-	1.0, 1.5, 2.0
-	PDCCF Range-3	201–300	201–290	5	8 × 16	-	-	1.0, 1.5, 2.0

.

To evaluate the performance of all the algorithms for detection, clutter rejection, and classification as well as tracking of the three objects in three-target sequence, the designed detection and classification filters of Table 9 for different ranges are applied to Sequence L1618. The results obtained for the EMACH-PDCCF algorithm with Sequence L1618 are shown for some sample frames in Figure 11. The tracking algorithm places “T1” for the detected Class-1 type targets (APC1), “TII” for the detected Class-2 type targets (M60) and “TIII” for the detected Class-3 type targets (truck) at their corresponding locations in the frames. In Figure 11, the frame numbers are shown at the lower left corner of each frame. The complete tracking results for the sequence are summarized in

Table 10. Tracking results of three-target sequences.

Seq. Name	Total Frames	No. of ROIs Taken	Threshold	Target Name	No. of Frames Target Present	No. of Frames Detected Correctly	Total No. of False Alarms	Percentage of Successful Detection
MACH-DCCF Algorithm
L1618	300	6	0.30, 0.52, 0.52	APC1	300	273	54	91
				M60	103	71	0	69
				truck	17	0	0	0
MACH-PDCCF Algorithm
L1618	300	6	0.52	APC1	300	297	20	99
				M60	103	99	0	96
				truck	17	12	0	71
EMACH-PDCCF Algorithm
L1618	300	4	0.52	APC1	300	297	23	99
				M60	103	99	0	96
				truck	17	12	0	71

for all algorithms. In the threshold column of the table, a single value indicates that a single threshold is used for all ranges. Otherwise, different threshold values in that column displays the thresholds used for various ranges of classification filters employed.

12. Conclusions

Pattern recognition and tracking in FLIR imagery is a challenging problem due to various factors such as low resolution, low signal-to-noise ratio, different 3D orientations of the targets, effects of global motion, and close proximity with similar objects. In this paper, we reviewed the recent trends and advancements in distortion-invariant pattern recognition algorithms for single/multiple target detection and tracking in FLIR imagery using correlation filters. Each detection/tracking algorithm utilizes various properties of targets and image frames of a given FLIR sequence. Test results using real life FLIR image sequences are presented to verify the effectiveness of the filter-based pattern recognition and tracking techniques. Future work in this area would include a review of techniques beyond correlation that are particularly useful for high resolution targets. Also, development and inclusion of techniques with a dynamic update of the target model may be considered.