Infrared Small Target Detection via Modified Random Walks

Abstract

Infrared small target detection under intricate background and heavy noise is one of the crucial tasks in the field of remote sensing. Conventional algorithms can fail in detecting small targets due to the low signal-to-noise ratios of the images. To solve this problem, an effective infrared small target detection algorithm inspired by random walks is presented in this paper. The novelty of our contribution involves the combination of the local contrast feature and the global uniqueness of the small targets. Firstly, the original pixel-wise image is transformed into an multi-dimensional image with respect to the local contrast measure. Secondly, a reconstructed seeds selection map (SSM) is generated based on the multi-dimensional image. Then, an adaptive seeds selection method is proposed to automatically select the foreground seeds potentially placed in the areas of the small targets in the SSM. After that, a confidence map is constructed using a modified random walks (MRW) algorithm to represent the global uniqueness of the small targets. Finally, we segment the targets from the confidence map by utilizing an adaptive threshold. Extensive experimental evaluation results on a real test dataset demonstrate that our algorithm is superior to the state-of-the-art algorithms in both target enhancement and detection performance.

1. Introduction

Infrared search and tracking (IRST) systems have been widely applied in remote sensing, space surveillance, external intrusion warnings [1,2,3], etc. For early warning applications, incoming targets such as missiles, aircraft, and boats are supposed to be detected at a long distance. Due to the remotely sensed imaging, the targets in an infrared image are of small size with a lack of prior knowledge about the target shape and texture features [4,5]. Moreover, the targets are usually buried in heavy noise and complicated background clutter (e.g., irregular sunlit spot, sky–sea background, and heavy cloud). The above hostile situations make the signal-to-clutter ratio (SCR) of small targets very low [6]. Therefore, small target detection under heavy noise and complex background is considered to be a difficult and challenging problem. Although many research efforts have been made in this task over the past decades [7,8,9], it remains an open issue.

There are numerous methods for small-target detection problems, and these algorithms can be roughly categorized into two representative groups: track before detection (TBD) [10,11,12] and detection before track (DBT) [13,14,15]. Prior knowledge of the targets and backgrounds are necessarily required and have strong influence for the TBD methods. Furthermore, TBD methods are normally time-consuming because they involve the processing of sequential images. Compared to TBD methods, DBT methods only focus on detecting the small targets in a single frame, which makes them more time-saving and require less prior knowledge about the targets and backgrounds. Therefore, the DBT methods are more appropriate in some practical applications with real-time detection demands, and have recently attracted great attention [16,17,18].

Different kinds of DBT methods have been developed. Some conventional methods attempt to segment targets from the infrared images by spatial filtering and background estimation. For instance, Deshpande et al. [19] adopted Max-Mean and Max-Median filters to suppress the background clutter, Bai et al. [20] and Zhou et al. [21] applied a Top-Hat filter to small targets detection, and some other spatial filtering based methods (e.g., Laplacian of Gaussian (LoG) filter [22], two-dimensional least-mean square (TDLMS) filter [23], and bilateral filter (BF) [24,25]) have been proposed. These methods are easy to implement but are sensitive to noise and not eligible for scenes with varying backgrounds.

Motivated by the recent advances in the theory of the human visual system (HVS), many methods have been presented. Since an infrared dim-small target usually occupies only a few pixels and exhibits a signature of discontinuity with its neighboring regions rather than the global image, a number of HVS-based methods attempt to describe the local feature with different feature descriptors, for example, the local contrast measure (LCM) [26], the multiscale relative local contrast measure (RLCM) [27], and the weighted local difference measure (WLDM) [28]. Similar to the local feature descriptor-based methods, some entropy-based methods [29,30] and saliency-based methods [31,32] have also been presented. Although these approaches have shown better detection performance compared to conventional methods, they failed when the targets were embedded in thick cloud and heavy noise.

Based on the constructive work that Candes et al. [33] made about robust principal component analysis (RPCA), numerous methods have been proposed to detect infrared small targets. These methods consider that the target components and background components can be approximately represented by a sparse matrix and a low-rank matrix, respectively. For instance, Gao et al. [34] presented an infrared patch-image (IPI) model by transforming the the original infrared image into a raw matrix and then recovering the sparse matrix and low-rank matrix from it. Similarly, Wang et al. [35] generated an image patch set by multiscale patch transformation and achieved the matrix recovery using an adaptive weighting parameter. Additionally, other components analysis-based methods such as singular value decomposition [36] and partial sum minimization of singular values [37] have been proposed. Although their methods perform better than the conventional method for complex scenes, the process of component composition and reconstruction is expensive and time-consuming.

Inspired by statistics research and computer vision theory, Wang et al. [38] proposed a small target detection method utilizing least squares support vector machine (LS-SVM). Li et al. [39] employed the local steering kernel (LSK) to encode the infrared image patch and proposed a local adaptive contrast measure based on regularized LSK reconstruction. Moreover, Gao et al. [40] developed their IPI model by taking the temporal cue into consideration and presented a statistical model based on the mixture of Gaussians (MoG) and the Markov random field (MRF). In addition to the state-of-the-art studies, there are still many other algorithms for detecting dim-small targets, such as algorithms based on the statistical regression [9], manifold learning [41], the nearest neighbor classifier [42], neural networks [43], etc.

Notice that small targets reveal different patterns compared with the backgrounds, thus they can be taken as anomalies (or outliers) in the infrared images. In this case, small target detection can also be considered as anomaly detection. In the past decades, anomaly detection has been well studied and widely used in detecting small targets from multispectral and hyperspectral images. One of the best-known anomaly detection methods is the Reed–Xiaoli (RX) algorithm [44]. RX detector attempts to measure the difference of outliers and backgrounds with Mahalanobis distance. However, it is fragile when the feature of the outliers lies in a high-dimensional space. To solve this problem, Kwon et al. [45] proposed a nonlinear version of the RX-algorithm by using kernel functions. Later, Zhou et al. [46] developed a faster version of the kernel-based RX-algorithm by grouping background pixels into clusters in the preprocessing step.

The intention of this research was to propose a small target detection algorithm that is effective and efficient in detecting small targets in varying scenes and robust to noise pollution. Note that small targets are generally of small size and target birth causes remarkable changes of local textural characteristics. Moreover, the pixels of the background clutter are distributed over the entire image while the pixels of small targets are unique within the entire image. With these considerations in mind, we first construct a local contrast element for each pixel according to its local discontinuity with the neighboring pixels. We adaptively select potential foreground seeds and background seeds based on these elements. Then, the global uniqueness of the targets is represented by applying a modified random walks (MRW) algorithm. The contributions of this paper can be summarized as follows:

• A multi-dimensional image is constructed, and each element of the image delineates the local contrast of the corresponding pixel in the original infrared image. Additionally, the local contrast element is utilized as an indicator of the seeds selection for the random walks algorithm.
• A confidence map for the original image is achieved using the MRW algorithm. The targets of interest are well enhanced, while the background clutter and noise are greatly suppressed.
• A dim-small target detection algorithm is designed based on MRW. To demonstrate the validity of the algorithm, a real dataset is used in our experiments. The extensive experimental results show that the proposed method is superior to the state-of-the-art methods with respect to target enhancement performance and detection accuracy.

The remainder of this paper is organized as follows: Section 2 describes the proposed small target detection method based on MRW in detail. Section 3 gives comprehensive experimental results on the simulated and real datasets. Section 4 contains the target enhancement and detection performances on the four real sequences and the comparison with the current methods. Conclusions and perspectives are presented in Section 5.

2. Methodology

In this section, we first briefly review the theory of the random walks algorithm. Then, the specific mathematical description of the proposed local contrast element is given. Subsequently, an adaptive seeds selection method based on local contrast element is presented. Then, we provide an illustration of the modified random walks algorithm (MRW) with the selected seeds. Finally, we introduce a full framework and a detailed algorithm of the MRW-based small target detection method.

2.1. Overview of Random Walks

Random walks is a mathematical model of a random process, which leads an unlabeled element to a labeled seed with the highest likelihood [47]. Given a graph structure $G=(V,E)$ with nodes $v\in V$ and edges $e\in E$, the task of random walks is to group the nodes into K classes. Assume that the elements set is given as $\mathcal{X}=\left\{x_1,\cdots ,x_N\right\}\in {\mathbb{R}}^m$, where N is the number of elements. The connection between any two nodes $x_u$ and $x_v$ in $\mathcal{X}$ (i.e., the weight of the edge $w_{uv}$) is quantified by a weight matrix $W={\left[w_{uv}\right]}_{N\times N}$. Here, we assume that the connection between any two nodes is undirected (i.e., $w_{uv}=w_{vu}$) [47] and $w_{uv}$ is empirically obtained by the typical Gaussian weighting function:

$$w_{uv}=\exp (-\frac{\left|\right|I_{x_u}-I_{x_v}{\left|\right|}^2}{{\sigma }^2}),$$

(1)

where $I_{x_u}$ indicates the intensity of $x_i$ and $\sigma$ is a constant that controls the weighting degree. The degree of $x_u$ is given by

$$d_u=\sum _v{w}_{uv}.$$

(2)

Without loss of generality, we partition the elements set $\mathcal{X}$ into two sets ${\mathcal{X}}_M$ (marked/seed nodes) and ${\mathcal{X}}_U$ (unseeded nodes) such that ${\mathcal{X}}_M\cup {\mathcal{X}}_U=\mathcal{X}$ and ${\mathcal{X}}_M\cap {\mathcal{X}}_U=\varnothing$. The elements in ${\mathcal{X}}_M$ are classified into K categories according to the defined label function:

$$Q\left(x_u\right)=k,k\in \mathbb{Z},0<k<K.$$

(3)

Let $p^k={[p_1^k,\cdots ,p_N^k]}^T=[{\left(p_M^k\right)}^T,{\left(p_U^k\right)}^T]$ denote the probability vector of the elements in $\mathcal{X}$ for label k, where $p_M^k$ is for the labeled seeds, which has fixed value as

$$p_s^k=\left\{\begin{array}{cc}1\hfill & Q\left(x_s\right)=k,\hfill \\ 0\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(4)

for any $x_s\in {\mathcal{X}}_M$. We further define the $N\times N$ Laplacian matrix L as

$$L_{uv}=\left\{\begin{array}{cc}\hfill d_u& \hfill \mathrm{if}u=v,\\ \hfill -w_{uv}& \hfill \mathrm{if}{x}_u\mathrm{and}{x}_v\mathrm{are}\mathrm{adjacent}\mathrm{nodes},\\ \hfill 0& \hfill \mathrm{otherwise}.\end{array}\right.$$

(5)

Note that L is symmetric since the edges E are undirected. With all of the above preparation, the optimized $p^k$ is achieved by minimizing the decomposed Dirichlet integral [47]:

$$D\left[p^k\right]=\frac{1}{2}{\left(p^k\right)}^{T}L{p}^k=\frac{1}{2}\left[{\left(p_M^k\right)}^T{\left(p_U^k\right)}^T\right]\left[\begin{array}{cc}{L}_M& B\\ B^T& L_U\end{array}\right]\left[\begin{array}{c}{p}_M^k\\ p_U^k\end{array}\right].$$

(6)

The critical point is found by differentiating $D\left[p^k\right]$ with respect to $p_U^k$. That is,

$$p_U^k=-L_U^{-1}{B}^T{p}_M^k.$$

(7)

We were inspired by the conventional random walks algorithm, which is highly suitable for measuring the global uniqueness of small targets. Assume that we have found a set of candidate targets including both ground-true and -false targets, we assign K labels to them and group these labeled nodes into ${\mathcal{X}}_M$, and group the remaining pixels of the input image into an unlabeled nodes set ${\mathcal{X}}_U$. In addition, the local and global similarity of the pixels can be measured by the weight matrix W and the weighting degree $d_u$, respectively. After that, the small target detection is converted to find the labeled pixels that only few unlabeled pixels lead to, i.e., find the labeled nodes with sparse $p_U$. With these considerations in mind, we modified the random walks algorithm and designed a small target detection algorithm based on the modified random walks algorithm.

2.2. Adaptive Seeds Selection Based on Seeds Selection Map (SSM)

Since random walks has the ability to measure the global uniqueness of the labeled seeds, numerous researchers have made great progress by applying this model to target detection [48,49]. The performance of this model is highly dependent on the user-defined seeds. Therefore, it is important to select appropriate foreground and background seeds. In the following, we design an adaptive seeds selection method for the random walks algorithm.

2.2.1. Seeds Selection Map (SSM)

As mentioned previously, the small target reveals structure discontinuity with its neighboring pixels rather than the entire image. That is, the intensity of the target is greater (or lower) than the remnant pixels in a local region. From this view, we conceive a local contrast element to represent the local structure discontinuity of the target.

Let an input raw infrared image denoted by I with size $m\times n$ be defined on the global patch domain $\mathcal{I}\subset {\mathcal{R}}^2$ and a sub-patch P with size of $d\times d$ be defined on the patch domain $\mathcal{P}\subset \mathcal{I}\subset {\mathcal{R}}^2$, as shown in Figure 1a. As depicted in Figure 1b, the sub-patch ${\mathit{P}}_{(i,j)}$ is partitioned into $\lfloor d/2\rfloor$ ($\lfloor ·\rfloor$ is the floor function) parts such that ${\mathit{P}}_{(i,j)}={\bigcup }_{l=0}^{\lfloor d/2\rfloor }{\mathit{P}}_{(i,j)}^l$ and ${\mathit{P}}_{(i,j)}^l$ contains the neighboring pixels with Chebyshev distance of l from the center pixel $(i,j)$. That is, for any $l\in \{0,1,\cdots ,\lfloor d/2\rfloor \}$, we have

$${\mathit{P}}_{(i,j)}^l=\left\{(x,y)\left|max\right(|x-i|,|y-j|)=l\right\}.$$

(8)

Define

$$(x_{ij}^l,y_{ij}^l)=\underset{(x,y)\in {\mathit{P}}_{(i,j)}^l}{argmin}|I(i,j)-I(x,y)|,\forall l\in \{0,1,\cdots ,\lfloor d/2\rfloor \},$$

(9)

where $I(x,y)$ is the intensity of the pixel $(x,y)$ in I. That is, $(x_{ij}^l,y_{ij}^l)$ is the pixel in ${\mathit{P}}_{(i,j)}^l$ that has the minimum difference with the center pixel $(i,j)$. Let a vector ${\mathit{c}}_{ij}=[c_{ij}^1,\cdots ,c^{\lfloor d/2\rfloor }]$, the so-called local contrast element, describe the discontinuity structure between the center pixel $(i,j)$ and the remnant pixels in ${\mathit{P}}_{ij}$, and each element of ${\mathit{c}}_{ij}$ can be obtained by

$${\mathbf{c}}_{ij}^l=I(i,j)-I(x_{ij}^l,y_{ij}^l),\forall l\in \{1,\cdots ,\lfloor d/2\rfloor \}.$$

(10)

Assume that a seeds selection map (SSM) $\mathit{S}$ is defined on a domain $\mathcal{S}\subset {\mathcal{R}}^2$. Let the transformation function $\Psi :\mathcal{I}\to \mathcal{S}$ be defined as follows:

$$S(i,j)=\Psi \left(I(i,j)\right)=\sum _{l=1}^{\lfloor d/2\rfloor }\left|{\mathbf{c}}_{ij}^l\right|{\mathbf{c}}_{ij}^l.$$

(11)

As shown in Equation (11), $S(i,j)$ quantifies the “signed energy” of the local energy element ${\mathit{c}}_{ij}$, which is rational for practical applications since the pixels with continuous directed difference are more likely to be a target.

To demonstrate the validity of SSM, some SSM examples of representative infrared images are displayed in Figure 2. It can be seen in Figure 2 that the targets in these infrared images are buried in extremely complex background clutters and are barely distinguishable. After transforming the raw images into seeds selection maps, the background clutters are fairly suppressed and the targets are well enhanced. However, some pixels of irregular edges and heavy noise are also enhanced, which leads to a high false alarm rate. Thus, further processing is needed.

2.2.2. Adaptive Seeds Selection for Random Walks

As discussed previously, the validity of the user-defined seeds has a great effect on the performance of the random walks algorithm. Note that the SSM achieved in Section 2.2.1 greatly enhances the potential small targets, thus we can select seeds for random walks based on SSM. We define the thresholds $t_{fore}$ as follows:

$$t_{fore}=\mathrm{mean}\left(S\right)+\lambda \ast \mathrm{std}\left(S\right),$$

(12)

where $\lambda$ is an empirical constant that is discussed in Section 3. The threshold is utilized to select pixels with $S>t_{fore}$ as foreground seeds. For convenience of implementation of the random walks algorithm, the selected seeds with Chebyshev distances between each other within 5 are considered as one effective seed, which is placed in the seed with greatest value of S. Specifically, the progress of selecting seeds from SSM is as follows: (1) Given an SSM S, select candidate pixels and group them into a set $\mathit{T}=\left\{(x,y)\right|S(x,y)>t_{fore}\}$. (2) If there exists two pixels $(x_1,y_1)$, $(x_2,y_2)\in \mathit{T}$ with $max\left(\right|x_1-x_2|,|y_1-y_2\left|\right)\le 5$, remove the pixel with smaller S, e.g., if $S(x_1,y_1)<S(x_2,x_2)$, remove $(x_1,y_1)$ from T. (3) Repeat Step 2 until all redundant pixels are removed.

The next step is to apply the random walks model to assign the unlabeled pixels to the labeled seeds. As a first impression, the process of random walks is similar to clustering. However, they are different statistical models in many aspects. Here, we take the well-known k-means clustering [50] as an example of clustering model and summarize the major differences of these two models as follows:

• The goals of these two methods are different. Random walk searches the optimal path from one state to another state, that is, leads one state to another state with highest probability. In the proposed algorithm, the random walks model finds the optimal paths from unlabeled pixels to the labeled pixels and obtains the probabilities of walking unlabeled pixels to labeled pixels. On the other hand, k-means aims to group scattered samples to several clutters with respect to the similarity of samples.
• The random walk process is a kind of Markov process, that is, every state of the model is only allowed to be transited to its adjacent states. For example, if state i and state j are not adjacent, the transition probability of moving from state i to state j is 0. Therefore, the weight $w_{uv}$ of pixels $x_u$ and $x_v$ are set to 0 if they are not adjacent. However, the process of k-means clustering has no such limitation.
• The geographical information of states (pixels in the image) is involved in the random walks algorithm. The father the distance between two states i and j, the longer is the optimal path, thus the smaller is the probability of leading state i to state j. On the other hand, k-means does not consider the geographical information of the pixels.

2.3. Modified Random Walks (MRW)

Based on the seeds determined in Section 2.2.2, the seeded pixels in the input infrared image I correspond to the marked nodes in ${\mathcal{X}}_M$ (see Section 2.1). Assume that there are K seeds. The seeds are then combined into $p_M^k$ obtained by Equation (4), where $k\in \{1,\cdots ,K\}$ corresponds to the label of the kth seed. Since the small targets are strongly related to their local regions, we modify the Laplacian matrix of Equation (5) as

$$L_{uv}=\left\{\begin{array}{cc}\hfill d_u& \hfill \mathrm{if}u=v,\\ \hfill -w_{uv}& \hfill \mathrm{if}{x}_v\in {\mathit{P}}_{x_u},\\ \hfill 0& \hfill \mathrm{otherwise},\end{array}\right.$$

(13)

where L is an $N\times N$ matrix, $N=m\times n$ is the number of pixels in I, $w_{uv}$ is obtained by substituting the intensity of pixel $x_u$ and $x_v$ to Equation (1), and $d_u$ is calculated by Equation (2). Instead of measuring the connection between adjacent nodes (please refer to the second condition of Equation (5)), the modified Laplacian matrix measures the connection among the nodes belonging to the same local region. It is rational since the small targets usually occupy few pixels and highly relate to their surrounding pixels. Then, the probability vector $p_U^k$ of ${\mathcal{X}}_U$ for label k is computed according Equations (6) and (7).

As discussed previously, the pixels of the background clutter and noise are distributed all over the image and exhibit similar patterns. In contrast, pixels of the small targets are rather unique in the image and only relate to their neighboring pixels. Correspondingly, if the kth seed $x_k\in {\mathcal{X}}_M$ is a ground-true target, then there only exists few unseeded pixels with high probability in $p_U^k$. Otherwise, there will be numerous unseeded pixels with high probability in $p_U^k$. Briefly, $p_U^k$ is sparse if $x_k$ is a ground-true target, while it is non-sparse if $x_k$ is a false alarm. Therefore, we reconstruct a composite $\tilde{p}$ by taking the global uniqueness into consideration, and we have

$${\tilde{p}}_u=\underset{k\in \{1,\cdots ,K\}}{max}\frac{p_u^k}{{\sum }_{v=1}^N{p}_v^k},\forall x_u\in \mathcal{X}.$$

(14)

That is, the final probability ${\tilde{p}}_u$ of $x_u$ is considered as the maximal normalized probability of $x_u$. Then, we reshape $\tilde{p}$ to a matrix C with same size of the input image I as the final confidence map for small target detection.

The confidence maps of the representative infrared image in Figure 2 are displayed in Figure 3. As shown in Figure 3, the targets are greatly enhanced, while the background clutter and noise are perfectly suppressed in the map. The red circles indicate where the ground-true targets are.

2.4. Small Target Detection Algorithm Based on MRW

It can be seen in Figure 3 that the contrast between the dim-small targets and the background is greatly enlarged in the confidence map C obtained by MRW, which makes it easy to detect the target. Similar to Equation (12), we segment the target with an adaptive threshold defined by

$$t_{seg}=\mathrm{mean}\left(C\right)+\lambda \ast \mathrm{std}\left(C\right),$$

(15)

where $\lambda$ is the same constant in Equation (12).

Based on the aforementioned preparation, a small target detection algorithm is designed. The flowchart of the proposed method is depicted in Figure 4 and can be summarized as five steps: (1) Construct a multi-dimensional image with the input infrared image using the method in Section 2.2.1. As shown in Figure 4, there are totally $\lfloor d/2\rfloor$ layers of the transformed image and each layer has the same size as the input image. (2) Transform the multi-dimensional image into the seeds selection map according to Equation (11). (3) Select the foreground seeds with the seeds selection map according to Section 2.2.2. (4) Perform the modified random walks algorithm with the selected seeds and reconstruct a confidence map using the method in Section 2.3. (5) Segment the small targets from the confidence map by Equation (15). The detailed algorithm is presented in Algorithm 1, where d is the sub-patch size (see Figure 1a), $N=m\times n$ is the number of pixels in the input image, and K is the number of selected seeds.

Algorithm 1 Small target detection based on MRW

Input: Input image I

Output: Position of the target $(i_t,j_t)$

1: Initialization: $S={\mathbf{0}}_{m\times n}$, $C={\mathbf{0}}_{m\times n}$;

2: Construct the seeds selection map (SSM):

3: for $i=\lfloor d/2\rfloor +1:m$ do

4:     for $j=\lfloor d/2\rfloor +1:n$ do

5:         for $l=1:\lfloor d/2\rfloor$ do

6:            Construct a sub-patch ${\mathit{P}}_{(i,j)}^l$ according to Equation (8);

7:            Find the pixel in ${\mathit{P}}_{(i,j)}^l$ that has minimum difference with $(i,j)$, see Equation (9);

8:            The lth value of the local contrast element (i.e., ${\mathit{c}}_{ij}^l$) is calculated by Equation (10);

9:         end for

10:         The value of the pixel $(i,j)$ in the SSM (i.e., $S(i,j)$) is obtained by Equation (11).

11:     end for

12: end for

13: Adaptive seeds selection:

14: Select the pixels in I with $S>t_{fore}$ (refer to Equation (12) for $t_{fore}$) as the foreground seeds.

15: Modified random walks (MRW):

16: for $u=1:N$ do

17:     for $v=1:N$ do

18:         Construct the modified Laplacian matrix $L_{uv}$ according to Equation (13);

19:     end for

20: end for

21: for $k=1:K$ do

22:     Assign the label k to the kth foreground seed;

23:     The probability vector for the seeds $p_M^k$ is obtained by Equation (4);

24:     The probability vector for the unseeded pixels $p_U^k$ is computed by Equation (7);

25: end for

26: The composite $\tilde{p}$ is obtained by Equation (14).

27: Reconstruct $\tilde{p}$ to a confidence map C with the same size of the input image I.

28: Adaptive target segmentation:

29: Segment the target placed in $(i_t,j_t)$ with $C(i_t,j_t)>t_{seg}$ (refer to Equation (15) for $t_{seg}$).

3. Experimental Results

In this section, we first introduce the test dataset and the evaluation metrics. Then, we analyze the sensitivity of the crucial parameters of the proposed method and present the experimental results on the test dataset with the optimized parameters.

3.1. Dataset and Evaluation Metrics

In our experiment, three real consecutive infrared images sequences and a set of single frame infrared images were used. Hereinafter, these sequences are named Seq 1, Seq 2, and Seq 3, and the set of single frame images is referred to as Set R. There were, respectively, 30, 300, 100, and 26 images included in Seq 1, Seq 2, Seq 3 and Set R. Seq 1 records an aircraft flying through the heavy clouds. The aircraft is buried in the thick clouds in the first 15 frames. In Seq 2, an airplane is flying in the sky, and almost every image is damaged. A bird flying at low altitude is recorded in Seq 3, and the images are overexposed. Set R contains some representative single-frame infrared images with dim-small targets embedded in diverse backgrounds, and most of them are available in [51].

The performance of a small target algorithm is generally evaluated on the basis of three criteria: target enhancement, background suppression, and detection accuracy. The signal-to-clutter gain (SCRG) is widely used for evaluating the performance of target enhancement [40]. However, the clutter around targets is purely clean after using our method, which makes the SCRG approach infinity. Thus, we adopted the local contrast gain (LCG) to evaluate the target enhancement ability. The LCG is defined as:

$$\begin{array}{c}\hfill LCG=\frac{{\mathrm{LC}}_{out}}{{\mathrm{LC}}_{in}},\\ \hfill LC=\frac{|{\mu }_t-{\mu }_b|}{{\mu }_t+{\mu }_b},\end{array}$$

(16)

where $L{C}_{out}$ and $L{C}_{in}$ are the local contrast (LC) of the output and original images, respectively; and ${\mu }_t$ and ${\mu }_b$ are the mean gray levels of the target region and the neighboring regions, respectively. The performance of the background suppression was evaluated by the background suppression factor (BSF), defined as

$$BSF=\frac{{\sigma }_{out}}{{\sigma }_{in}},$$

(17)

where ${\sigma }_{out}$ and ${\sigma }_{in}$, respectively, denote the standard deviation of the whole output and original images.

In addition, the probability of detection $P_d$ and the false alarm rate $P_f$ were utilized to evaluate the detection accuracy of the proposed method, and are expressed as [16]:

$$\begin{array}{c}\hfill P_d=\frac{N_{true}}{N_{act}},\\ \hfill P_f=\frac{N_{false}}{N_{img}},\end{array}$$

(18)

where $N_{true}$, $N_{act}$, $N_{false}$ and $N_{img}$ denote the numbers of true detections, actual targets, false detections and frames, respectively.

3.2. Sensitivity Analysis of Crucial Parameters

Basically, there are three parameters involved in our algorithm: the size of the sub-patch P (see d in Figure 1a), the weight constant $\lambda$ in Equations (12) and (15), and the controlling parameter $\sigma$ of Equation (1). The sub-patch size d determines the dimension of the local contrast element $\mathit{c}$, and thus indirectly determines the validity of the seeds selection map. Additionally, d is involved in constructing the modified Laplacian matrix (see Equation (13)). The weight constant $\lambda$ is mainly used for seeds selection and target segmentation. Though large $\lambda$ can suppress more clutter and noise, it may fail in selecting the seed placed in the target. We found that the proposed algorithm is effective for detecting small targets when weight constant $\lambda$ varies in a wide range of $[3,12]$. The controlling parameter $\sigma$ controls the weighting degree of the connection between any two nodes, and it produced satisfactory results when we selected $\sigma$ in an interval $[0.2,0.45]$. Here, we set $\lambda =5$ and $\sigma =0.2$ in our implementation.

The evaluated performance on the three sequences and Set R is reported in

Table 1. Average background suppression factor (BSF) and local contrast gain (LCG) of the three sequences and Set R processed by the proposed modified random walks (MRW)-based small target detection algorithm with variation of the sub-patch size d.

		d = 3	d = 5	d = 7	d = 9	d = 11	d = 13
Seq 1	$\overline{LCG}$	8.4383	11.8134	12.3235	13.2478	13.2818	12.9601
	$\overline{BSF}$	15.2645	17.9574	19.6027	19.6489	18.2879	18.2440
Seq 2	$\overline{LCG}$	10.4703	10.8178	15.1055	15.8983	15.9998	16.0171
	$\overline{BSF}$	53.5936	50.5754	57.2290	55.6125	53.9464	52.9188
Seq 3	$\overline{LCG}$	6.7753	6.8763	7.2176	7.3431	7.3740	7.3768
	$\overline{BSF}$	54.3929	61.5580	72.5739	74.3424	73.8086	72.9750
Set R	$\overline{LCG}$	5.0603	5.9314	6.0667	6.0954	6.0439	6.0453
	$\overline{BSF}$	16.0908	19.5771	21.0352	20.3556	19.4511	18.6230

The bolder data means the maximum of indicators.

and Figure 5. It can be seen in Table 1 that the proposed algorithm was effective for target enhancement and background suppression under various background conditions if the sub-patch size d was a member of {7, 9, 11, 13}. The receiver operating characteristic (ROC) curve is a graphical plot of the true detection rate $P_d$ against the false alarm rate $P_f$, and is popularly used for intuitively evaluating the validity of target detection algorithms. ROC curves for different sub-patch sizes are displayed in Figure 5. With sub-patch size d set to 11 or 13, the proposed algorithm had better ROC performance for the test images than d of 7 or 9. Additionally, we can learn from the performances of LCG, BSF, and ROC curves that our algorithm is moderately sensitive to the sub-patch size, since we represent the local structure of the small target by a local contrast element instead of the raw image patch. For comprehensive consideration, we set the sub-patch size d to 11.

3.3. Analysis of Robustness to Noise

Since the small targets are photographed under various intricate scenes by the thermal cameras, it is inevitable that the infrared images are polluted by noise, which results in rather low SCR of the small targets in the infrared images. Therefore, whether a small target detection method is robust to heavy noise is quite an important issue. To demonstrate the robustness to noise of the proposed MRW-based algorithm, we artificially added zero-mean Gaussian white noise to the input images and tested our algorithm on them, as shown in Figure 6. We repeatedly tested our method on the noised-added images 50 times;

Table 2. Mean local contrast ($\overline{LC}$) of the images in Figure 6.

	Raw Image	Confidence Map	Noise-Added Image	Confidence Map
Figure 6a	0.1120	0.9716	0.1001	0.8543
Figure 6b	0.1360	0.9863	0.1278	0.9562
Figure 6c	0.1491	0.9895	0.0992	0.9685
Figure 6d	0.0920	0.9776	0.0837	0.9603

contains the information of the mean LC (defined by Equation (16)) of the unprocessed and processed images in Figure 6. According to Table 2, the intensity difference between the targets and their neighboring backgrounds of the raw images was rather low. It became even worse after we added Gaussian white noise to them. However, the proposed MRW-based method could greatly enhance the targets despite the heavy noise. Most of the clutters and noise were suppressed, as seen from the three-dimensional meshes of the confidence maps in Figure 6. This indicates that our method is insensitive to noise.

4. Comparison and Discussion

To demonstrate the superiority of the proposed small target detection method based on modified random walks, several state-of-the-art methods were adopted for comparison. Specifically, some representative and advanced methods were chosen as baseline methods, such as the Top-Hat transformation-based (THT) method [20], the multiscale patch-based contrast measure-based (MPCM) method [52], the non-negative infrared patch-image model based on partial sum minimization of singular values-based (NIPPS) method [37], the gradient direction diversity weighted multiscale flux density-based (GDD-MFD) method [6] and the local steering kernel (LSK) reconstruction-based method [39]. For comprehensive comparison, we discuss the performances of these methods in three respects: target enhancement, background suppression, and detection accuracy. Section 4.1 presents the target enhancement performance and background suppression performance evaluated by LCG and BSF, respectively. The detection performance on the real dataset of different methods are discussed in detail in Section 4.2. The computational complexity of the proposed algorithm is analyzed in Section 4.4. All experiments were implemented using MATLAB R2018a on a computer with a 3.6 GHz Intel core i7 CPU and 8 GB RAM.

4.1. Comparison of Target Enhancement and Background Suppression

The key issue of the small target detection method is the transformation of a raw infrared image into a confidence map in which the target region is well-enhanced while the background and the noise are eliminated. An advanced small target detection method produces a precise map which makes the target readily detected, thus leading to a low $P_f$ with high $P_d$. More than 450 real infrared images were tested by using the proposed MRW-based method and the baseline methods. There were various small targets embedded in different kinds of backgrounds, including the backgrounds sky–sea, heavy cloud, country road, daytime forest, etc.

Visual results of the samples processed by different methods are shown in Figure 7. For fair comparison, the output maps of all the methods were normalized to $[0,1]$. It can be seen in Figure 7 that the Top-Hat-based method could eliminate a large amount of clutters. However, it mistakenly enhanced the strong edges, as seen from Figure 7a2,f2. The MPCM-based method was effective for enhancing small targets embedded in rather homogeneous background. Nevertheless, it failed in eliminating strong edges such as the heavy clouds in Figure 7a3,b3, the horizontal line of the sky-ground in Figure 7c3, and the oblique line of the country road in Figure 7f3. Likewise, the NIPPS-based and the GDD-MFD-based methods revealed similar drawbacks when strong edges and heterogeneous cluttersexist; specifically, the NIPPS-based method was quite sensitive to strong edges (see Figure 7a4–c4) and the GDD-MFD-based method performed poorly in clutter suppression (see Figure 7a5–f5). By contrast, the LSK-based method could eliminate most of the clutter and the edges. However, LSK may fail in detecting small targets when the infrared image is of low quality. For example, this method incorrectly eliminated the small target in Seq 2 since the images in Seq 2 were obtained by an infrared sensor of poor signal-to-noise ratio (SNR) under atrocious weather (see Figure 7b6). Compared with the above baseline methods, the proposed method could robustly suppress more clutter and noise and accurately enlarged the contrast of the target and background. The same conclusion can be made from the results in

Table 3. Average LCG and BSF indicators of the output maps using different methods on the test dataset.

		THT	MPCM	NIPPS	GDD-MFD	LSK	MRW (Ours)
Seq 1	$\overline{LCG}$	12.7690	1.6411	11.5069	3.2155	13.9788	13.2818
	$\overline{BSF}$	3.5883	4.0456	3.4685	3.4350	17.0702	18.2879
Seq 2	$\overline{LCG}$	2.8119	1.7078	11.3092	2.5913	12.6941	15.9998
	$\overline{BSF}$	19.2082	7.3354	7.3544	6.7750	43.1069	53.9464
Seq 3	$\overline{LCG}$	7.1285	1.1596	6.1982	1.6439	7.0014	7.3740
	$\overline{BSF}$	25.5728	12.4371	9.2242	9.6839	62.7293	73.8086
Set R	$\overline{LCG}$	4.7732	1.4484	4.2882	1.1869	5.7011	6.0954
	$\overline{BSF}$	5.2039	6.3063	4.8676	4.7243	17.6723	21.0352

The bolder data means the maximum of indicators.

. It can be seen in Table 3 that the output maps of the proposed MRW-based method had the greatest value of the LCG and BSF indicators for most images. It is noteworthy that the advanced LSK-based method [39] performed better than the other baseline methods, and even had a similar value of LCG and BSF to our method for Seq 1. However, the LSK-based method was sensitive to the bright damaged point, as depicted in Figure 7b6 and was more time-consuming than our method (as discussed in Section 4.4). In summary, our method noticeably outperformed the baseline methods in both target enhancement and background suppression and was more robust to the varying scenes.

4.2. Comparison of Detection Performances

To compare the detection ability of the methods, the ROC curves obtained by different methods for the dataset were utilized as the indicator. For fair comparison, the detected results were considered a positive result if the pixel distance between the center of the ground-true target and the detected result was less than 4 pixels, as suggested in [28].

As depicted in Figure 8, the proposed MRW-based method performed better in detection accuracy compared with the baseline methods. It is notable that our method always had the highest $P_d$ under the same $P_f$ for Seq 2, Seq 3 and Set R. Specifically, there are a total of 44 ground-true targets in Seq 1, and 14 of them are barely distinguishable. As shown in Figure 8a, the THT-based, NIPPS-based, GDD-MFD-based and MRW-based methods could detect 30 targets without generating any false alarm. In addition, except for the NIPPS-based and MRW-based methods, the $P_d$ hardly reached 0.727 (32 ground-true targets correctly detected) when $P_f$ exceeded 10 (300 false alarms generated) for the other methods, which is unacceptable. Moreover, when $P_f\le 20$, our method owned the highest $P_d$ under the same $P_f$. For Seq 2, the LSK-based method was ineffective in detecting the targets due to the low quality of the infrared images. On the other hand, when $P_d$ reached 0.99 (i.e., 297 of 300 targets were detected), the $P_f$ of our method was 5.75 while the $P_f$ of the suboptimal THT-based method exceeded 7.5. For Seq 3, the MRW-based method showed the best detection performance and reached $P_d=1$ when $P_f=0.01$. That is, there was only one false alarm generated when all the ground-true targets were correctly detected. In contrast, the NIPPS-based and LSK-based methods showed bad detection performances for the dim and small targets embedded in the overexposed background. Different from the image sequences Seq 1, Seq 2, and Seq 3, Set R contains some single-frame infrared images with small targets buried in diverse complex and noisy backgrounds, and there are a total of 31 targets in the 26 frames. Thus, it is convincing that a small target detection method is robust to different backgrounds if it shows great detection performance for Set R. As shown in Figure 8d, the proposed MRW-based method had the best detection performance for Set R. It worth noting that the proposed method could correctly segment 29 targets with only one false alarm and achieved Pd = 1 with only five false alarms. By way of comparison, the LSK-based method had the suboptimal performance and reached $P_d=1$ when $P_f=4.6923$, and the other baseline methods showed poor robustness for Set R. The comparison results in Figure 8 suggest that the proposed MRW-based method is superior to the baseline methods for detecting the dim and small targets against different background and heavy noise.

4.3. Comparison with RX-Related Methods

As discussed previously, RX-related anomaly detection methods have proven effective for detecting small targets from multispectral and hyperspectral images. Since the single-band infrared image is transformed into a multi-dimensional image after the first two steps in the proposed algorithm (see Figure 4), it is operable to directly apply these anomaly detection algorithms to the transformed images. We tested local RX-algorithm (LRX) [44], local kernel RX-algorithm (LKRX) [45] and cluster kernel RX-algorithm (CKRX) [46] on the seeds selection maps of the dataset. The ROC curves of the RX-related algorithms and the proposed algorithm are presented in Figure 9. As shown in Figure 9, our method outperformed the RX-related methods with respect to detection accuracy. RX-detector is highly sensitive to intensity inhomogeneity, and it had the worst performances for Seq 1, Seq 3 and Set R due to the existence of strong edges and clutters in the images. Compared with LRX, LKRX and CKRX could suppress more clutters by using kernel functions. However, they were sensitive to abnormal pixels, e.g., dead pixels in Seq 2. Moreover, the RX-related methods were much more time-consuming than the proposed method, as shown in

Table 4. Average running time (s) of different methods testing on the dataset.

	THT	MPCM	NIPPS	GDD-MFD	LSK	LRX	LKRX	CKRX	MRW (Ours)
Seq 1	0.0429	3.3479	3.1729	2.7777	25.5441	20.7095	119.8360	13.3303	0.4288
Seq 2	0.0303	5.2819	9.5939	4.4714	43.1069	34.6333	134.4858	31.0786	0.7543
Seq 3	0.0313	7.6518	12.8935	4.1875	66.1112	49.7072	148.9765	18.7797	0.9256
Set R	0.0442	1.6208	1.6079	1.0004	12.8281	6.1094	29.4660	4.2343	1.2653

. In summary, our method is superior to the RX-related methods both in detection ability and algorithm efficiency.

4.4. Computational Complexity

To show the efficiency of the proposed algorithm, we analyzed the computational complexity of the proposed algorithm and compared the running time of various methods testing on the dataset. The major computation cost of our method arises from two parts: the reconstruction of SSM and the inference of the MRW model. According to Algorithm 1, in each iteration, it costs $O\left(d^2\right)$ time to construct the local contrast element for each sub-patch with size of d, and it is the major contribution to the computational cost of the first part. As for the later part, the modified Laplacian matrix can be constructed in $O\left(N^2\right)$ time and the optimization problem can be solved in $O\left(N^3\right)$ time, thus it costs about $O\left(N^3\right)$ time to infer the MRW model. In summary, the computation complexity of the proposed method is around $O(N{d}^2+N^3)$ when there are total N pixels in the input image. The average running time of different methods testing on the dataset is reported in Table 4, from which we can see that the THT-based method has the highest computation speed for the test dataset and LKRX is the most time-consuming method. Except for the THT-based method, the proposed method consumed the least running time for Seq 1, Seq 2 and Seq 3, and had acceptable computation speed for Set R.

5. Conclusions

It is important that small target detection algorithms are robust and efficient to meet the requirements of early warning systems. However, due to the long-distance imaging, the targets in infrared images are of small size and usually with quite low signal-to-noise ratio, which makes the detection task very difficult. To solve this problem, this paper presents a small target detection method based on modified random walks. Different from the state-of-the-art small target detection algorithms, we treat the dim and small target as a salient target with little prior knowledge. There are roughly two stages in our algorithm: First, the candidate targets are determined according to the seeds selection map (SSM), which is constructed by using the so-called local contrast element. Second, the authenticity of the candidate targets are verified based on the modified random walks and adaptive segmentation. Extensive experiments on a real dataset suggest that our algorithm is robust and effective for small target detection regardless of the varying backgrounds and the poor image quality. The computation of the proposed algorithm mainly comes from the second stage, more specifically, the evaluation of the matrix inversion in Equation (7). Since the Laplacian matrix (see Equation (13)) is a sparse and real symmetric matrix, the evaluation of the matrix inversion can be less time-consuming with the help of the matrix optimization theory. Moreover, we note that the proposed algorithm is highly suited for parallel implementation. Thus, the computing speed can be further increased by using a Graphic Processing Unit (GPU) or Field-Programmable Gate Array (FPGA). In the future, we will continue working on the improvement of the current algorithm and investigating a faster version.