Tracking and Data Association Based on Reinforcement Learning

Abstract

Currently, most multi-target data association methods require the assumption that the target motion model is known, but this assumption is clearly not valid in a real environment. In the case of an unknown system model, the influence of environmental clutter and sensor detection errors on the association results should be considered, as well as the occurrence of strong target maneuvers and the sudden appearance of new targets during the association process. To address these problems, this paper designs a target tracking and data association algorithm based on reinforcement learning. First, this algorithm combines the dynamic exploration capability of reinforcement learning and the long-time memory function of LSTM network to design a policy network that predicts the probability of associating a point with its various possible source targets. Then, the Bayesian network and the multi-order least squares curve fitting method are combined to predict the location of target, and the results are fed into the Bayesian recursive function to obtain the reward. Simultaneously, some corresponding mechanisms are proposed for possible problems that interfere with the association process. Finally, the simulation experimental results show that this algorithm associates the results with higher accuracy compared to other algorithms when faced with the above problem.

1. Introduction

Data association is a key technique within the field of radar data processing. The technique is used to determine the correct track by establishing relationships between radar measurements at adjacent moments. In general, the traditional data association algorithms are centered on the predicted value of track and select the qualified points according to some specific criteria. These algorithms are generally related to filtering algorithms in target tracking. For example, Probability Data Association Filter (PDAF), Joint Probability Data Association Filter (JPDAF), Multiple Hypothesis Tracking Filter (MHTF), Interacting Multiple Model Filter (IMMF), and Probabilistic Hypothesis Density Filter (PHDF) [1,2,3,4,5]. Except for the PDAF algorithm, other algorithms can be used to handle the problem of multi-target data association.

Radar measurement data can be affected by many factors, such as sensor detection errors and clutter generated by the external environment. Additionally, the motion mode of target can increase the difficulty of data association. To solve these problems, scholars have proposed many related algorithms [6,7,8,9]. For example, a truncated JPDAF algorithm is proposed in reference [3], which can effectively filter the clutter in the data combined with the characteristics of target motion. In reference [4], fuzzy recursive least squares filtering is combined with JPDAF to achieve effective tracking of maneuvering targets in the environment where both the measurement error and the motion model are uncertain. In reference [5], the consistent distributed filtering process was incorporated into the JPDAF algorithm using the random Gibbs sampling algorithm to improve the accuracy of association results. In reference [6], a radiometric intensity PHDF algorithm was designed to solve the tracking problem of close targets. In reference [7], the DBSCAN algorithm and the sequential random sample consistency algorithm are used to preprocess the measurement data, which substantially reduces the computational resources of algorithm. In reference [8], a new multi-sensor PHDF algorithm is proposed that can fuse information from different sensors and overcome the difficulties of missing the priori statistical information such as clutter, measurement noise, and detection probability. In some specific environments, by some special means, these algorithms are able to handle some multi-target data association problems. However, none of them has completely escaped the limitation of traditional filter framework, and they need to realize the data association of multiple targets based on the system model. Obviously, the system model in the real environment is unpredictable, which is one of the reasons why the traditional filter algorithms are not very practical. Therefore, how to achieve accurate association of multiple targets in the complex environment where the priori information is scarce is an urgent problem.

In recent years, with the development of artificial intelligence, using artificial intelligence to analyze data has become the mainstream in the new era [10,11,12]. Similarly, there are many scholars who have incorporated artificial intelligence techniques into the data association process and have achieved some results. Compared with traditional filter algorithms, these results have some breakthroughs in some specific environments. However, in the final analysis, they still follow the traditional data association rules and combine with filter to process data. In this case, it must be assumed that the system model is known. However, in the real environment, the system model is difficult to predict in advance, so it is worthwhile to study how to solve the problem of target tracking and data association without the priori conditions.

Reinforcement Learning (RL) is an important branch in the field of machine learning. It emphasizes how to act based on the environment to obtain the maximum expected benefits. After decades of development, RL technology has many achievements, such as Q-learning, dynamic programming, Policy Gradients, Deep-Q-Network, etc. [13,14,15,16,17,18,19,20,21,22,23,24]. In essence, RL is a process in which the agent learns itself in an unknown environment under defined rules. The behavior of agent conforms to the real environment. The reward or punishment are fed by defined rules. The end mark is the arrival of “terminus “. In simple terms, the target data association process is to find the real points generated by the track, and all the points are sorted according to the time axis to form the complete track. This process is not only similar to finding the optimal path but can also be roughly regarded as a snake game. The environment of target requires the agent to adapt. The rules of the game are determined by the motion of target. Therefore, in theory, RL can solve the problem of target tracking and data association.

In the real environment, the target system model is unknown, and the association results are vulnerable to environmental clutter, detection errors of sensors, strong maneuvers, and the sudden appearance of new targets. Traditional data association algorithms have difficulty in achieving data association of multiple targets within the constraints of these conditions. In this paper, a novel tracking and data association network architecture based on RL networks is established. First, a policy network that can predict the probability of associating the point with its various possible source targets is designed, which exploits the dynamic exploration capability of RL techniques and the long-time memory function of LSTM networks. Then, the multi-order least square curve fitting method is used to predict the position of point. The order of least square curve fitting method is obtained by Bayesian network analysis. The results are wired into the Bayesian recursive function to obtain the reward for each point. Finally, some corresponding mechanisms are proposed for the characteristics of targets and measurements. For example, the ring wave gate mechanism that eliminates some of the effects of clutter, the detection mechanism that enables timely detection of new targets, the Compete mechanism that solves multiple target association problems, and the adjustment mechanism that automatically corrects the association results. This paper pioneers a data association idea of association while learning to achieve accurate association of measurement and target. This has high practical engineering significance and helps national defense construction, which is a trend of future development.

The paper is organized as follows. Section 2 introduces the overall network architecture of algorithm and briefly explains the learning approach of the network. Section 3 designs the state space and agent detection mechanism according to the motion trend of target. Section 4 designs the Ring Wave Gate Screening mechanism and the Compete mechanism, taking into account the characteristics of measurement. Section 5 defines the reward function and details the design ideas. Section 6 proposes an adaptive adjustment mechanism for the problems that may occur in the complex and variable multi-target data association process. Section 7 trains the network with a large amount of simulation data, tests the performance of algorithm, and makes the corresponding analysis. Section 8 summarizes the advantages and disadvantages of this algorithm and looks forward to the future development direction.

2. Network Architecture

In the process of tracking and data association, sensors continuously obtain measurement data in a certain time sequence, which contains both real points of target and clutter generated by sensor interference, external environment, and other factors. When at a certain moment, some of the clutter may be mixed with the points generated by target, and it is difficult to accurately associate the points originating from the target. Therefore, this paper designs a tracking and data association network architecture based on RL, as shown in Figure 1.

First, this architecture can output the association probability of targets and points in a real environment lacking a priori information using a policy network and select the point with the highest probability for association. When there are multiple agents associated with points in the detection area at the same time, competition may occur, so the Compete mechanism is set to let each agent make a suitable choice. Then, the reward function calculates the reward of current selection and feeds back the evaluation of policy network. Finally, the optimal policy that matches the real motion of target is found by learning to achieve accurate association between target and point. The network architecture contains four core components, namely Environment State Design, Action Selection, Reward Function Definition, and Policy Network Learning.

3. Environment State Design

3.1. Agent Definition

All moving targets have inertia, and inertia makes the motion trend of adjacent time targets similar. Based on the motion trend, agent can find the points originating from the target and achieve accurate association between measurement and target. A single point cannot be used to represent the motion trend of a target, and multiple consecutive moments are needed to approximate the motion trend of a target. Therefore, a “sliding window” state is designed in this paper. The state consists of association results for $N$ consecutive moments. $N$ is the “window”, and the value of $N$ is determined by the sampling interval of sensor. In general, the larger value of the sampling interval is, the smaller value of $N$ is.

As shown in Figure 1, $Z_{t-N}$ is the set of measurements when $t-N$, and $s_n^{\mathrm{t}}$ is the state consisting of points in $\{Z_{t-1},Z_{t-2},\dots ,Z_{t-N}\}$ originating from the target $n$ when $t$. Suppose $z_i^{t-N}=\left[\begin{array}{c}{x}_i^{t-N}\\ y_i^{t-N}\end{array}\right]$ is the point in $Z_{t-N}$. $x_i^{t-N}$, and $y_i^{t-N}$ are the positions of the x and y axes in the Cartesian coordinate, respectively. If $s_n^{\mathrm{t}}$ consists of $\{z_i^{t-1},z_i^{t-2},\dots ,z_i^{t-N}\}$, the expression of $s_n^{\mathrm{t}}$ is as follows.

$$s_n^{\mathrm{t}}=\left[\begin{array}{c}{X}_n^t\\ Y_n^t\end{array}\right]=\left[\begin{array}{cccc}{x}_i^{t-1}& x_i^{t-2}& \dots & x_i^{t-N}\\ y_i^{t-1}& y_i^{t-2}& \dots & y_i^{t-N}\end{array}\right]$$

(1)

3.2. Agent Detection

According to Equation (1), it can be seen that each state is composed of points associated with $N$ consecutive moments. However, at the beginning of data association process, the real information of agent is unknown, and the initial state of agent needs to be obtained from the measurement data of previous $N$ moments. Similarly, during the data association process, some new agents may appear in the detection area at any time, and their real information is obtained from the measurement data. Therefore, an agent detection mechanism is designed in this paper. Considering that randomly distributed clutter is difficult to be associated into a complete track at $N$ consecutive moments, this mechanism defines that the primary condition for determining the appearance of a new agent is the presence of points originating from the target at $N$ consecutive moments. The flow of this mechanism is shown below.

• An exhaustive enumeration is used to traverse the measurement data at $N$ moments to find all possible new agents. Based on the sampling interval $T\_sample$ of the sensor, the measurement data $z$, the maximum velocity $v\_max$, and the minimum velocity $v\_min$ of target motion, the points at adjacent moments are required to meet the velocity threshold.

  $$v\_min\le \frac{‖\overrightarrow{z_i^{t-N}{z}_i^{t-N+1}}‖}{T\_sample}\le v\_max$$

  (2)

As shown in the above equation, the calculation of $z_i^{t-N}$ and $z_i^{t-N+1}$ is used as an example. It can be calculated for all points at adjacent moments and thus filter all states of agent that meet the velocity threshold requirements.

2.

The agents are grouped according to the principle of “a target corresponds to a point and a point originates from a target”. Usually, in the process of multi-target data association, by default a target can only generate one point, and similarly, a point can only originate from one target. Based on this principle, all the identified agents are grouped to ensure that all agents do not select multiple points at the same time, and that a point is not selected by multiple agents in a group at the same time.

3.

If the agent already exists, the groups that are similar to the already existing agent are deleted according to the Cyclic Frechet Distance Clustering method. As shown in Figure 2, $S\_new$ denotes the set of all agent groups; $S_i$ denotes the state set of the $i$-th group of agents in $S\_new$; $s_j$ denotes the state of the $j$-th agent in $S_i$; $S\_survival$ denotes the state set of already existing agent; and $s_z$ denotes the state of the $z$-th agent in $S\_survival$. First, in order to compare the similarity of states $s_j$ and state $s_z$ more conveniently, the states $s_j$ and state $s_z$ are normalized. Then, $Frechet(s_j,s_z)$ is calculated according to reference [25]. Finally, the value domain of $Frechet(s_j,s_z)$ can be controlled at (0, 1). As shown in Figure 2, $d$ is the basis for judging whether states $s_j$ and state $s_z$ are similar, and $d=0.5$ is usually set. If the new agent still exists in $S\_new$, the next step is continued. If not, the Agent Detection mechanism ends.

4.

The motion track of target in the environment can be represented by the Vector Space Model, where two points at adjacent moments form a vector. Three points at consecutive moments constitute two vectors, and the cosine of angle between these two vectors is assumed to be the motion trend factor of target during this time. The motion trend factor of track at three consecutive moments can be calculated according to the cosine theorem formula.

$$f_i^{t-N+1}=\frac{\overrightarrow{z_i^{t-N}{z}_i^{t-N+1}}•\overrightarrow{z_i^{t-N+1}{z}_i^{t-N+2}}}{‖\overrightarrow{z_i^{t-N}{z}_i^{t-N+1}}‖‖\overrightarrow{z_i^{t-N+1}{z}_i^{t-N+2}}‖},f_i^{t-N+1}\in \left[-1,1\right]$$

(3)

According to the above equation, the motion trend $f=\left\{f_i^{t-2},f_i^{t-3},\dots ,f_i^{t-N+1}\right\}$ of a new agent at $N$ consecutive moments can be calculated, and thus the motion trend of all new agents can be obtained.

5.

Consider that variance can be used to measure the magnitude of fluctuations in a batch of data. With the same sample size, the larger the variance is, the more volatile and unstable the data are. For the track, the more stable the motion trend is, the higher its reliability is. Therefore, the variance of $f$ is obtained as

$$Variance=\mathrm{var}(f)$$

(4)

Finally, the data with the smallest $Variance$ from each category is found as the state of new agent, and the category is the number of new agents.

4. Action Selection

In the tracking and data association process, the agent needs to associate the points from the measurement data that are most likely to originate from the target. Such points are the actions that need to be selected, and the points in consecutive moments constitute the action space. As shown in Figure 1, the policy network can output the association probability of state and measurement, and the agent selects the action based on the association probability. Considering that clutter is randomly distributed, the spatial distance between some of the clutter and the current moment position of agent may far exceed the motion limit of agent in a single sampling interval. Therefore, a Ring Wave Gate Screening process is designed. In addition, a Compete mechanism is designed to solve the competition problem that may occur when there are multiple agents selecting actions simultaneously in the detection area.

4.1. Ring Wave Gate Screening

The points originating from the target necessarily follow the motion rules of target. The ring distribution area of points can be determined based on the maximum velocity $v\_max$ and minimum velocity $v\_min$ of target motion, and all points that may be generated by target can be found.

As shown in the Algorithm 1, the inputs are the measurement data $Z$ at the current moment, the maximum velocity $v\_max$, the minimum velocity $v\_min$, the position $z$ of target at the current moment, and the sampling interval $T\_sample$ of sensor. $z_i$ is the $i$-th point of $Z$. $x_i$ and $y_i$ are the positions of $z_i$ in the x-axis and y-axis, respectively. $Z$ has the total of $n$ points. $x$ and $y$ are the positions of $z$ in the x-axis and y-axis, respectively. The output is the filtered measurement data $Z\_$. The mechanism needs to calculate the distance between all points and $z$. Then, all points that meet the velocity threshold are filtered out.

Algorithm 1. Pseudo code for Ring Wave Gate Screening.
Input: Z = {zi = [xi, yi]|i = 1, 2,…, n}, v_max, v_min, z = [x,y], T_sample
Output: Z_
1:	i = 1
2:	I = []
3:	while i <= n do
4:	d = ((x − xi)2 + (y − yi)2)1/2
5:	if d >= v_min ∗ T_sample and d <= v_max ∗ T_sample then
6:	I.append(i)
7:	end if
8:	i = i + 1
9:	end while
10:	Z_ = {zi|i = I}
11:	return Z_

However, considering the effect of sensor detection error ${\sigma }_v$, the speed has to vary with the detection error so that

$$v\_max=v\_max+\frac{{\sigma }_v^2}{T\_sample}$$

(5)

$$v\_\min =v\_\min -\frac{{\sigma }_v^2}{T\_sample}$$

(6)

As shown in Figure 3, target 1 moves in the detection area, and there are nine points in the measurement data at the next moment. By drawing the ring wave gate area, it can be seen that only four points may originate from the target, namely $z_1$, $z_2$, $z_3$, and $z_4$.

The Ring Wave Gate Screening process uses basic information about target and sensor to remove some of the clutter from the measurement data. This not only increases the likelihood of selecting points originating from the target but also saves the computational resources of policy network.

4.2. Compete Mechanism

When there are multiple agents in the detection area, the association probability of agent and point can be obtained through the policy network. In this case, an agent will select the point with the highest association probability. However, if multiple agents select points at the same time, competition is bound to occur.

Suppose two targets in the detection area cross meet, and there are nine points in the measurement data at the next moment. The ring wave gate areas of target 1 and target 2 are shown in Figure 3 and Figure 4, respectively. Target 1 has four points that may originate from the target, namely $z_1$, $z_2$, $z_3$, and $z_4$. Target 2 also has four points that may originate from the target, namely $z_1$, $z_3$, $z_4$, and $z_6$. Obviously, when faced with $z_1$, $z_3$, and $z_4$, there are competitions between target 1 and target 2.

In the face of competing phenomena, the association probability matrix is generated based on the state of current moment and the measurement data of next moment.

$$P^t=\left[\begin{array}{c}{P}_1^t\\ P_2^t\\ ⋮\\ P_n^t\end{array}\right]=\left[\begin{array}{cccc}{p}_{11}^t& p_{12}^t& \dots & p_{1m}^t\\ p_{21}^t& p_{22}^t& \dots & p_{2m}^t\\ \begin{array}{c}⋮\end{array}& \begin{array}{c}⋮\end{array}& \begin{array}{c}\end{array}& \begin{array}{c}⋮\end{array}\\ p_{n1}^t& p_{n2}^t& \dots & p_{nm}^t\end{array}\right]$$

(7)

In the above equation, $P^t$ is the association probability matrix when $t$; $P_n^t$ is the association probability matrix of the agent $n$ with measurement data when $t$; and $p_{nm}^t$ is the association probability of the agent $n$ with point $m$ when $t$.

First, the clutters for each agent are found based on the ring wave gate. Then, in the association probability matrix, the association probability of each agent with its clutter is set to 0, and the association probability of each agent with other points is output by the policy network. Finally, the agent needs to follow two rules when choosing the action:

• Each point has only a unique source target.
• A target can generate at most one point.

Based on these two rules, the point with the highest association probability is selected for each agent. If multiple agents select the same points, the points are assigned according to the following priority.

• The agent has only one point that may originate from the target.
• The association probability between agent and point is maximum.

According to this way, all agents can be associated to points.

Through the Compete mechanism, it is possible to match the most appropriate action to each agent and find the point that originates from the target.

5. Reward Function Definition

This paper defines the reward function that can calculate the true score of the action selected by agent. First, this function predicts the order of least squares by Bayesian network based on the state of the agent at the current moment. Then, it uses least squares to predict the position of target at the next moment. Finally, it is combined with the Bayesian recursive function to calculate the reward for the action selected by the agent. As shown in Figure 1, the reward is directly input into the loss function of LSTM network, which can feed back the evaluation of policy network and help the agent find the optimal policy that matches the real motion of target.

In the real environment, the motion of the target is complex and variable. A complete track may contain many kinds of motion models such as Constant Velocity (CV), Coordinate Turn (CT), and Constant Acceleration (CA). If the least squares method of fixed order is used, it cannot accurately describe the motion trend of the whole track. Therefore, it is necessary to predict the order of least squares that can fit the motion trend of target at these $N$ moments based on the current state. Referring to the Bayesian network model in reference [26], this paper designs a Bayesian network with $M$ classification function. The value of $M$ indicates that the order of least squares is divided into $M$ classes. If the value of $M$ is too small, it is difficult to fit all possible motion trends. If the value of $M$ is too large, the fitting results are prone to large deviations. The input of network is the state of agent at the current moment. The output of network is the probability of each class. The class with the highest probability is chosen as the order of least squares method.

Then, according to the predicted order $g$, this function uses least squares [27] to fit the state of agent and predict the position at the next moment. The fitted state is $\widetilde{s_n^{\mathrm{t}}}=\left[\begin{array}{c}\widetilde{X_n^t}\\ \widetilde{Y_n^t}\end{array}\right]$. The predicted position is $\widetilde{z_n^t}=\left[\begin{array}{c}\widetilde{x_n^t}\\ \widetilde{y_n^t}\end{array}\right]$.

$$\widetilde{X_n^t}=M_t{W}_x$$

(8)

$$\widetilde{Y_n^t}=M_t{W}_y$$

(9)

$$\widetilde{x_n^t}=M_{t+1}{W}_x$$

(10)

$$\widetilde{y_n^t}=M_{t+1}{W}_y$$

(11)

In the above equation, $W_x$ and $W_y$ are the fitted coefficient matrices.

$$W_x={(M_t{}^T{M}_t)}^{-1}{M}_t{}^T{X}_n^t$$

(12)

$$W_y={(M_t{}^T{M}_t)}^{-1}{M}_t{}^T{Y}_n^t$$

(13)

$M_t$ is the fitted information when $t$.

$$M_t=[T_t^0,T_t^1,\dots ,T_t^g]$$

(14)

$$T_t^g=\left[t-1^g,t-2^g,\dots ,t-N^g\right]$$

(15)

Finally, referring to the Bayesian recursive function in reference [7], the reward function $r_n^t$ is defined as

$$r_n^t=\frac{P\_D{r}_n^{t-1}{q}_n^t(z_i^t)}{K_t(Z_{}^t)+P\_D{r}_n^{t-1}{q}_n^t(z_i^t)}$$

(16)

$$q_n^t(z_i^t)=N(z_i^t;\widetilde{z_n^t},S_n^t)$$

(17)

$$S_n^t=\mathrm{cov}(z_i^t,\widetilde{z_n^t})+R$$

(18)

In the above equation, $z_i^t$ is the point $i$ selected from the measurement data $Z_{}^t$ when $t$. $K_t(Z_{}^t)$ is the clutter intensity when $t$, namely $K_t(Z_{}^t)=\frac{nu{m}_{Z_{}^t}}{TS}$. $nu{m}_{Z_{}^t}$ is the number of points in the measurement data $Z_{}^t$. $TS$ is the area of sensor detection region. $R$ is the covariance matrix of measurement, which is also the detection error of sensor.

6. Policy Network Learning

As shown in Figure 1, the policy network of this network architecture uses the LSTM network [28]. The input layer of network is the state of the agent at the current moment and a point at the next moment. Since the output of policy network is the association probability between state and point, and the range of association probability is $\left[0,1\right]$, a Sigmoid layer is added after the output layer of LSTM network.

6.1. Learning Style

In the real environment, there are factors such as high maneuverability of target and dense clutter interference. It is difficult to learn the motion trend of multiple targets simultaneously, and the accuracy cannot be guaranteed. Usually, the multi-target data association process can be regarded as the multiple single-target data association processes. Therefore, based on the learning outcomes of a single target, supplemented by an adaptive adjustment mechanism, the precise association of multiple targets can be achieved.

As shown in Figure 5, the single-target learning process consists of two parts, namely training and testing. The whole process has modules such as initialization, ring wave gate screening, action selection, and stepping. After single-target learning is completed, multi-target testing is conducted based on the learning experience. The multi-target testing process is basically the same as the single-target learning process, except that Compete mechanism and adaptive adjustment mechanism are added.

In Figure 5, Random Choice module and Argmax Choice module are action selection processes. In the Random Choice module, the state of agent and a randomly selected point are input into the policy network to obtain the association probability. In the Argmax Choice module, the state of agent and all points are input into the policy network to obtain all association probabilities, and the point with the highest probability is selected for testing. The Random Choice module enables the learning of strategies in the continuous trial-and-error process, while the Argmax Choice module can be used as the means of testing the results of learning. Done is the marker that determines whether the learning process is ended or not. There is no clear “terminal” for the data association process. When there is no new measurement data at the next moment, the learning process is considered to be ended, namely Done is “True”. The Discount Sum Rewards module is the process of calculating the cumulative return of track discount. The Normalization module is to normalize the cumulative return of track discount. The Save Data module is used to save the data. The Loss module is the cross-entropy loss function. The Optimizer module uses the Adam optimization algorithm with the learning rate of 0.01.

6.2. Adaptive Adjustment Mechanism

In the real environment, there are many ways of target movement, and it is not possible for an agent to learn all of them. When testing a new strategy, the agent still needs to adapt to the environment, and it is difficult to accurately select the points originating from the target. Therefore, this paper designs a mechanism that can automatically adjust the motion trend during the test, which improves the weak migration ability of RL technique, as shown in Figure 6. When the test process ends, if the measurements are still present at the next moment, it indicates that there may be some incorrect association. This mechanism can find the wrongly selected actions and correct them.

As shown in Figure 6, this process can be performed in four steps.

• Find the minimum reward $r\_min$ and the corresponding state $s\_min$ from the reward set $reward$.
• Calculate the reward $r_i$ for each valid action $a_i(i=1,2,\dots ,m)$ according to Equation (16).
• Judgment: If the maximum reward $r\_max>r\_min$, the action is selected and the test continues. Conversely, remove $r\_min$ from $reward$ and continue the calculation.
• Iterate through all rewards in $reward$. If all rewards do not need to be adjusted, the association results are considered to have been associated without problems, and the test is over.

During multi-target data association testing, this mechanism can automatically adjust the track to make the association results more accurate. It also eliminates the process of re-adapting to the environment, making the algorithm more practical.

7. Simulation Experiment and Result Analysis

7.1. Simulation Environment Settings

Suppose there are flying targets in motion in an air-to-air radar detection area, and the sampling interval $T\_sample$ of system is 1, and the “window” $N$ is 5. The detection probability $\mathrm{P}\_\mathrm{D}$ of target is 1. The initial positions are randomly distributed at (−500 m, 500 m), and the initial velocity is randomly selected between 30 m/s and 100 m/s. The maximum speed of target is $v\_max=150 \mathrm{m}/\mathrm{s}$, and the minimum speed of target is $v\_min=10 \mathrm{m}/\mathrm{s}$. The number of clutter follows Poisson distribution, and the mean of Poisson distribution is $\lambda$. The acceleration in the CA model is randomly chosen between −10 m/s² and 10 m/s². The turning angular acceleration in the CT model is randomly chosen between 0 and 0.8.

According to reference [1], the motion process of target satisfies the state transition equation.

$${\mathit{X}}_k=\mathit{F}{\mathit{X}}_{k-1}+\Gamma v_{k-1}$$

(19)

In the above equation, $\mathit{F}$ is the state transition matrix, and $\Gamma$ is the process noise distribution matrix. $v_{k-1}$ is additive white noise, and its covariance matrix is ${\mathit{Q}}_{\mathit{k}-1}$. Combined with the parameters related to the real motion of target in the experiment, set ${\mathit{Q}}_{\mathit{k}-1}=\mathrm{diag}\left(\left[5^2,5^2\right]\right)$.

Corresponding to the CV model, CA model, and CT model, according to reference [1], the expressions of $\mathit{F}$ and $\Gamma$ are as follows:

$${\mathit{F}}_{CV}=\left[\begin{array}{cccc}1& T& 0& 0\\ 0& 1& 0& 0\\ 0& 0& 1& T\\ 0& 0& 0& 1\end{array}\right]{\Gamma }_{CV}=\left[\begin{array}{cc}\frac{T^2}{2}& 0\\ T& 0\\ 0& \frac{T^2}{2}\\ 0& T\end{array}\right]$$

$${\mathit{F}}_{CA}=\left[\begin{array}{cccccc}1& T& \frac{T^2}{2}& 0& 0& 0\\ 0& 1& T& 0& 0& 0\\ 0& 0& 1& 0& 0& 0\\ 0& 0& 0& 1& T& \frac{T^2}{2}\\ 0& 0& 0& 0& 1& T\\ 0& 0& 0& 0& 0& 1\end{array}\right]{\Gamma }_{CA}=\left[\begin{array}{cc}\frac{T^2}{2}& 0\\ T& 0\\ 1& 0\\ 0& \frac{T^2}{2}\\ 0& T\\ 0& 1\end{array}\right]$$

$${\mathit{F}}_{CT}=\left[\begin{array}{cccc}1& \frac{\sin \omega T}{\omega }& 0& \frac{\cos \omega T-1}{\omega }\\ 0& \cos \omega T& 0& -\sin \omega T\\ 0& \frac{1-\cos \omega T}{\omega }& 1& \frac{\sin \omega T}{\omega }\\ 0& \sin \omega T& 0& \cos \omega T\end{array}\right]{\Gamma }_{CT}=\left[\begin{array}{cc}\frac{T^2}{2}& 0\\ T& 0\\ 0& \frac{T^2}{2}\\ 0& T\end{array}\right]$$

According to reference [1], the measurement satisfies the following equation.

$${\mathit{Z}}_k=\mathit{H}{\mathit{X}}_k+{\mathit{W}}_k$$

(20)

In the above equation, $\mathit{H}$ is the measurement matrix. ${\mathit{W}}_k$ is white Gaussian noise, and its covariance matrix is ${\mathit{R}}_k=\mathrm{diag}\left(\left[\begin{array}{cc}{\sigma }_v^2& {\sigma }_v^2\end{array}\right]\right)$. The measurement error ${\sigma }_v^{}$ is determined by sensor performance.

Corresponding to the CV model, CA model, and CT model, according to reference [1], the expressions of $\mathit{H}$ are as follows:

$${\mathit{H}}_{CV}=\left[\begin{array}{cc}1& \begin{array}{ccc}0& 0& 0\end{array}\\ 0& \begin{array}{ccc}0& 1& 0\end{array}\end{array}\right]{\mathit{H}}_{CA}=\left[\begin{array}{cccccc}1& 0& 0& 0& 0& 0\\ 0& 0& 0& 1& 0& 0\end{array}\right]{\mathit{H}}_{CT}=\left[\begin{array}{cc}1& \begin{array}{ccc}0& 0& 0\end{array}\\ 0& \begin{array}{ccc}0& 1& 0\end{array}\end{array}\right]$$

7.2. Result Analysis

The whole experiment is divided into the single-target data association learning process and the multi-target data association testing process. The movement time of target is 30 s in the single-target data association learning process. The number of data in the training set is 5 groups. The measurement error is ${\sigma }_v^{}=0,10,20,30,40$, and each measurement error corresponds to one group of data. Each group has 10000 data, and the clutter parameter $\lambda$ is randomly selected from 0 to 90. The motion model of each track consists of any combination of the CV model, CA model, and CT model. The number of data in the testing set also is 5 groups. The measurement error is ${\sigma }_v^{}=0,10,20,30,40$, and each measurement error corresponds to one group of data. Each group has 100 data. The clutter parameter is $\lambda =0,10,20,\dots ,90$, and each clutter parameter has 10 data. The motion model of each track consists of any combination of CV model, CA model, and CT model.

After the test, the association accuracy $P$ can be calculated by comparing the test results with the true points originating from the target, namely $P=\frac{N_{correct}}{N_{all}}$ ($N_{correct}$ represents the number of true points originating from the target in the association result of agent, and $N_{all}$ represents the number of real points originating from the target, here $N_{all}=30$). After the test, the average association accuracy $\overline{P}$ can be calculated, namely $\overline{P}=\frac{1}{N_{data}}{\displaystyle \sum _{i=1}^{N_{data}}{P}_i}$ ($N_{data}$ represents the amount of data in the test set, here $N_{data}=10$). The experimental results of test are shown in Algorithm 1.

As shown in the

Table 1. Test results for single-target learning process.

$\overline{\mathit{P}}$	${\mathit{\sigma }}_{\mathit{v}}^{}=0$	${\mathit{\sigma }}_{\mathit{v}}^{}=10$	${\mathit{\sigma }}_{\mathit{v}}^{}=20$	${\mathit{\sigma }}_{\mathit{v}}^{}=30$	${\mathit{\sigma }}_{\mathit{v}}^{}=40$
$\lambda =0$	100	100	100	100	100
$\lambda =10$	100	100	99.67	98.33	95.67
$\lambda =20$	100	99	99.33	96.33	98.67
$\lambda =30$	100	99	94	94.67	88.67
$\lambda =40$	100	96	93.67	94.33	97.33
$\lambda =50$	100	93.33	92.67	90	95.33
$\lambda =60$	100	91	92.33	89.33	91
$\lambda =70$	100	90.33	92	87	86
$\lambda =80$	100	88.67	91.33	86.67	83.67
$\lambda =90$	100	83.33	90.67	84.33	80.33

, the average association accuracy is inversely proportional to measurement error and clutter parameters in terms of the general trend, and the values of $\overline{P}$ are affected by both. However, the value of $\overline{P}$ exceeds 80% in all cases, and there are only a few cases where the value of $\overline{P}$ is below 90%. Therefore, the test results of single-target learning process are relatively good and can basically assist in implementing the multi-target data association process.

The movement time of the target is 30 s in the multi-target data association testing process. The number of data in the testing set is 5 groups. The measurement error is ${\sigma }_v^{}=0,10,20,30,40$, and each measurement error corresponds to one group of data. Each group has 10 data, and the clutter parameter is $\lambda =0,10,20,\dots ,90$. The motion model of each track consists of any combination of CV model, CA model, CT1 model, CT2 model, and CT3 model. Figure 7 is the real track where target 5 appears suddenly at the fifth sampling interval. The state transition matrix and process noise distribution matrix of CV model and CA model are unchanged, namely ${\mathit{F}}_{CV}$, ${\Gamma }_{CV}$, ${\mathit{F}}_{CA}$, and ${\Gamma }_{CA}$. However, ${\mathit{F}}_{CT}$ and ${\Gamma }_{CT}$ of the CT model are also basically unchanged; only $\omega$ in ${\mathit{F}}_{CT}$ changes, and the corresponding relationship is

$$\mathrm{CT}1 \mathrm{model}: {\omega }_1=0.3 \mathrm{CT}2 \mathrm{model}: {\omega }_2=-0.3 \mathrm{CT}3 \mathrm{model}: {\omega }_3=0.5$$

Here, set the abbreviation of the algorithm in this paper as TDA. Figure 8, Figure 9, Figure 10, Figure 11 and Figure 12 show the measurement and association result of TDA at ${\sigma }_v^{}=0,\lambda =90$, ${\sigma }_v^{}=10,\lambda =80$, ${\sigma }_v^{}=20,\lambda =70$, ${\sigma }_v^{}=30,\lambda =60$, and ${\sigma }_v^{}=40,\lambda =50$, respectively. In the figure of measurement, the blue dots are measurements, and Target 1–Target 5 are the real points originating from the target. In the figure of association as result of TDA, the red dots are the real points originating from the target, and the green lines are the association track.

As shown in the following figure, when two targets have similar motion trends, close spatial distances, and close locations of real points for $N$ consecutive moments, there may be the “wrong selection”, and the agent may choose to associate the clutter if the real measurement at the next moment is close to the spatial location of the clutter. However, from the association results, these problems do not affect the association process in the subsequent moments and have the minimal impact on the whole process. There are two reasons to guarantee the performance of TDA. First, the fundamental method of TDA to achieve multi-target data association is to select the point that best matches the target motion trend from the next moment of measurement based on the motion trend of $N$ consecutive moments. An incorrectly associated point has less impact on this track, much less on the whole association process. Second, if the association process is interrupted due to the wrong association of some points, TDA can correct the error in time through the Adaptive Adjustment Mechanism to ensure the accuracy of association results. So, TDA can basically accurately associate to the points originating from the targets, and there are no problems of association interruptions, missed targets, and incorrect associations. It can effectively solve the data association problem of multiple targets.

In order to evaluate the performance of the algorithm more conveniently, several metrics are proposed in this paper. Assuming that the number of real points originating from the target is $num$, the number of all points in the measurement data is $num\_z\_all$, and the number of all real points originating from the target in the association result of agent is $num\_all$, the average overall association accuracy $P\_all$ is

$$P\_all=\frac{num\_all}{num}$$

(21)

The average association accuracy $P\_ztrue$ is

$$P\_ztrue=\frac{num\_all}{num\_z\_all}$$

(22)

The average accuracy of association results for each agent is $P\_equal$. Assuming that the association result of an agent corresponds to the track of a real target, if the number of real points originating from the target in the association result is $num\_i$, and the number of real points originating from the target is $num\_zi$, then

$$P\_equa{l}_i=\frac{num\_i}{num\_zi}$$

(23)

Set the number of targets in the association result to $num\_obs$.

Table 2. Change of $num\_all$.

$\mathit{n}\mathit{u}\mathit{m}\_\mathit{a}\mathit{l}\mathit{l}$	${\mathit{\sigma }}_{\mathit{v}}^{}=0$	${\mathit{\sigma }}_{\mathit{v}}^{}=10$	${\mathit{\sigma }}_{\mathit{v}}^{}=20$	${\mathit{\sigma }}_{\mathit{v}}^{}=30$	${\mathit{\sigma }}_{\mathit{v}}^{}=40$
$\lambda =0$	245	245	243.62	244.76	244.3
$\lambda =10$	244.83	244.25	243	243.88	243.01
$\lambda =20$	244.68	245	243.39	243.32	241.67
$\lambda =30$	244.97	245	242.52	242.54	240.33
$\lambda =40$	244.45	244.41	242.73	241.46	239.4
$\lambda =50$	244.26	244.43	242.28	240.86	238.47
$\lambda =60$	244.12	244.16	241.57	240	237.84
$\lambda =70$	244.11	244.28	241.46	239.28	237.03
$\lambda =80$	244	244.05	241.24	239.24	232.36
$\lambda =90$	243.76	244.08	241.67	238.54	234.52

,

Table 3. Change of $P\_all$.

$\mathit{P}\_\mathit{a}\mathit{l}\mathit{l}$	${\mathit{\sigma }}_{\mathit{v}}^{}=0$	${\mathit{\sigma }}_{\mathit{v}}^{}=10$	${\mathit{\sigma }}_{\mathit{v}}^{}=20$	${\mathit{\sigma }}_{\mathit{v}}^{}=30$	${\mathit{\sigma }}_{\mathit{v}}^{}=40$
$\lambda =0$	1	1	0.9944	0.9990	0.9971
$\lambda =10$	0.9993	0.9969	0.9918	0.9954	0.9918
$\lambda =20$	0.9987	1	0.9934	0.9931	0.9864
$\lambda =30$	0.998	1	0.9899	0.9900	0.9809
$\lambda =40$	0.9978	0.9976	0.9907	0.9856	0.9771
$\lambda =50$	0.9970	0.9977	0.9889	0.9831	0.9733
$\lambda =60$	0.9964	0.9966	0.9860	0.9796	0.9708
$\lambda =70$	0.9964	0.9971	0.9856	0.9767	0.9675
$\lambda =80$	0.9959	0.9961	0.9847	0.9765	0.9484
$\lambda =90$	0.9949	0.9962	0.9864	0.9729	0.9572

,

Table 4. Change of $num\_zall$.

$\mathit{n}\mathit{u}\mathit{m}\_\mathit{z}\mathit{a}\mathit{l}\mathit{l}$	${\mathit{\sigma }}_{\mathit{v}}^{}=0$	${\mathit{\sigma }}_{\mathit{v}}^{}=10$	${\mathit{\sigma }}_{\mathit{v}}^{}=20$	${\mathit{\sigma }}_{\mathit{v}}^{}=30$	${\mathit{\sigma }}_{\mathit{v}}^{}=40$
$\lambda =0$	245	245	243.63	244.96	244.54
$\lambda =10$	245	244.5	243.37	244.94	244.47
$\lambda =20$	244.97	245	244.06	244.74	244.47
$\lambda =30$	244.97	245	243.52	244.97	244.54
$\lambda =40$	244.96	244.97	243.8	244.77	244.45
$\lambda =50$	244.93	244.99	243.56	245.02	244.11
$\lambda =60$	244.87	245	243.25	244.9	244.38
$\lambda =70$	244.94	245	243.78	243.52	243.61
$\lambda =80$	244.85	245	243.66	243.68	238.31
$\lambda =90$	244.86	244.92	244.77	243.78	244.53

,

Table 5. Change of $P\_ztrue$.

$\mathit{P}\_\mathit{z}\mathit{t}\mathit{r}\mathit{u}\mathit{e}$	${\mathit{\sigma }}_{\mathit{v}}^{}=0$	${\mathit{\sigma }}_{\mathit{v}}^{}=10$	${\mathit{\sigma }}_{\mathit{v}}^{}=20$	${\mathit{\sigma }}_{\mathit{v}}^{}=30$	${\mathit{\sigma }}_{\mathit{v}}^{}=40$
$\lambda =0$	1	1	1	0.9992	0.9990
$\lambda =10$	0.9993	0.9990	0.9985	0.9957	0.9940
$\lambda =20$	0.9988	1	0.9972	0.9942	0.9886
$\lambda =30$	0.9981	1	0.9959	0.9901	0.9828
$\lambda =40$	0.9979	0.9977	0.9956	0.9865	0.9793
$\lambda =50$	0.9973	0.9977	0.9947	0.9830	0.9769
$\lambda =60$	0.9966	0.9966	0.9931	0.9800	0.9732
$\lambda =70$	0.9966	0.9971	0.9905	0.9826	0.9730
$\lambda =80$	0.9965	0.9961	0.9901	0.9818	0.9750
$\lambda =90$	0.9955	0.9966	0.9873	0.9777	0.9591

,

Table 6. Change of $P\_equal$.

$\mathit{P}\_\mathit{e}\mathit{q}\mathit{u}\mathit{a}\mathit{l}$	${\mathit{\sigma }}_{\mathit{v}}^{}=0$	${\mathit{\sigma }}_{\mathit{v}}^{}=10$	${\mathit{\sigma }}_{\mathit{v}}^{}=20$	${\mathit{\sigma }}_{\mathit{v}}^{}=30$	${\mathit{\sigma }}_{\mathit{v}}^{}=40$
$\lambda =0$	0.8320	0.8893	0.9114	0.9164	0.8827
$\lambda =10$	0.8351	0.9348	0.9286	0.9149	0.8569
$\lambda =20$	0.8438	1	09318	0.9159	0.8587
$\lambda =30$	0.8444	0.9036	0.9225	0.9218	0.8747
$\lambda =40$	0.8458	0.9374	0.9323	0.9113	0.8738
$\lambda =50$	0.8558	0.9318	0.9300	0.9070	0.8726
$\lambda =60$	0.8634	0.9312	0.9180	0.9044	0.8613
$\lambda =70$	0.8584	0.9381	0.9099	0.8775	0.8222
$\lambda =80$	0.8632	0.9447	0.9133	0.8749	0.7298
$\lambda =90$	0.8700	0.9525	09529	0.8871	0.8635

and

Table 7. Change of $num\_obs$.

$\mathit{n}\mathit{u}\mathit{m}\_\mathit{o}\mathit{b}\mathit{s}$	${\mathit{\sigma }}_{\mathit{v}}^{}=0$	${\mathit{\sigma }}_{\mathit{v}}^{}=10$	${\mathit{\sigma }}_{\mathit{v}}^{}=20$	${\mathit{\sigma }}_{\mathit{v}}^{}=30$	${\mathit{\sigma }}_{\mathit{v}}^{}=40$
$\lambda =0$	5	5	5.31	5.03	5
$\lambda =10$	5	5	5.16	5.01	5
$\lambda =20$	5	5	5.18	5.02	5.01
$\lambda =30$	5	5	5.26	5	5
$\lambda =40$	5	5	5.21	5	5.01
$\lambda =50$	5	5	5.19	5	5.06
$\lambda =60$	5	5	5.25	5.01	5.1
$\lambda =70$	5	5	5.29	5.23	5.26
$\lambda =80$	5	5	5.28	5.24	6
$\lambda =90$	5	5	5	5.13	5.04

show the performance of TDA when ${\sigma }_v^{}=0,10,20,30,40$ and $\lambda =0,10,20,\dots ,90$. Monte Carlo simulation is 100 times. Table 2 and Table 3 show the change of $num\_all$ and $P\_all$, respectively. As can be seen from the figure, when ${\sigma }_v^{}$ is larger, $P\_all$ is more affected by clutter. As $\lambda$ gets bigger, $P\_all$ gets smaller. However, the lowest of $P\_all$ remains at 0.95, which shows the validity of association results.

Table 4 and Table 5 show the change of $num\_zall$ and $P\_ztrue$, respectively. As shown in the figure, when ${\sigma }_v^{}$ is larger, $P\_ztrue$ is more affected by clutter. As $\lambda$ gets bigger, $P\_ztrue$ gets smaller. However, the lowest of $P\_ztrue$ is 0.96. So, the $P\_ztrue$ is high and stable.

Table 6 and Table 7 show the change of $P\_equal$ and $num\_obs$, respectively. As can be seen from the table, $P\_equal$ is stable at about 0.9, which is slightly affected by clutter and has good association performance. However, combined with $P\_all$ and $P\_ztrue$, $P\_equal$ decreases with the increase in ${\sigma }_v^{}$. This indicates that there may be a little crossover, aliasing, and other phenomena in the association tracks when the target is moving in parallel or hedging. However, the decrease in $P\_equal$ is not large, which proves that the association performance is still very good. The $num\_obs$ is stable. It is only at ${\sigma }_v^{}=40,\lambda =80$ that a large error occurs, and $num\_obs$ is only 6. This indicates that there are few interruptions in the association track.

The performance of this algorithm can be verified by combining the data in Table 2, Table 3, Table 4, Table 5, Table 6 and Table 7. First, the Adaptive Adjustment Mechanism plays the positive role to ensure the continuity of association track. Second, TDA can basically learn the real motion trend of the target, which ensures the accuracy of association results. Third, the agent detection mechanism can detect new targets that appear suddenly in the environment in time. Even if the association process is interrupted, it can still find the target and restart the association in the subsequent association process. In summary, although the performance of TDA decreases when ${\sigma }_v^{}$ and $\lambda$ increase, the results show that the performance of TDA decreases insignificantly, and the validity of association results can still be guaranteed. It is also proved that TDA can effectively handle the problems of data association for strongly maneuvering targets, even when multiple targets doing special maneuvers are also unaffected. Moreover, TDA does not require any impractical prerequisites and meets the requirements for solving practical problems.

In this paper, the TDA algorithm is compared with the JPDAF algorithm, MHTF algorithm, Fuzzy C-Means Clustering Filter (FCMF) algorithm [29], and Generalized Multi-Bernoulli Filter (GLMBF) algorithm [30]. Considering that most targets are maneuvering in the environment, the IMM model is added to the original algorithm, namely IMM-JPDAF algorithm, IMM-MHTF algorithm, IMM-FCMF algorithm, and IMM-GLMB algorithm. In the following, the four algorithms are described by their abbreviations, namely JPDA, MHT, FCM, and GLMB. Given the shortcomings of the four algorithms, it is necessary to assume that the initial position, motion model, and starting time of all target are known during the experiment. However, the TDA algorithm does not have this priori information.

Table 8. Performance comparison of $num\_all$ when ${\sigma }_v^{}=30$.

$\mathit{n}\mathit{u}\mathit{m}\_\mathit{a}\mathit{l}\mathit{l}$	JPDA	MHT	FCM	GLMB	TDA
$\lambda =0$	231	232.64	236.95	81.07	244.76
$\lambda =10$	221.5	232.8	237.24	63.04	243.88
$\lambda =20$	221.5	232.7	235.33	58.36	243.32
$\lambda =30$	244.1	232.98	235.53	53.88	242.54
$\lambda =40$	243.7	233.14	233.67	53.88	241.46
$\lambda =50$	230	233.22	233.57	50.29	240.86
$\lambda =60$	203.8	234.46	232.74	59.26	240
$\lambda =70$	245	233.86	231.32	62.05	239.28
$\lambda =80$	225.9	234.52	231.02	58.16	239.24
$\lambda =90$	228.3	234	230.9	53.08	238.35

,

Table 9. Performance comparison of $P\_all$ when ${\sigma }_v^{}=30$.

$\mathit{P}\_\mathit{a}\mathit{l}\mathit{l}$	JPDA	MHT	FCM	GLMB	TDA
$\lambda =0$	0.9429	0.9496	0.9671	0.3309	0.9990
$\lambda =10$	0.8816	0.9502	0.9683	0.2573	0.9954
$\lambda =20$	0.9041	0.9498	0.9605	0.2382	0.9931
$\lambda =30$	0.9963	0.9509	0.9613	0.2199	0.9900
$\lambda =40$	0.9947	0.9516	0.9538	0.2199	0.9856
$\lambda =50$	0.9388	0.9519	0.9533	0.2053	0.9831
$\lambda =60$	0.8318	0.9570	0.9500	0.2419	0.9796
$\lambda =70$	1	0.9545	0.9442	0.2533	0.9767
$\lambda =80$	0.9220	0.9572	0.9429	0.2374	0.9765
$\lambda =90$	0.9318	0.9551	0.9424	0.2167	0.9729

,

Table 10. Performance comparison of $num\_zall$ when ${\sigma }_v^{}=30$.

$\mathit{n}\mathit{u}\mathit{m}\_\mathit{z}\mathit{a}\mathit{l}\mathit{l}$	JPDA	MHT	FCM	GLMB	TDA
$\lambda =0$	231	232.64	245	178.5	244.96
$\lambda =10$	642.4	301.28	245	162	244.94
$\lambda =20$	1078.2	389.38	245	158.1	244.74
$\lambda =30$	1660.4	499.88	245	153.8	244.97
$\lambda =40$	2127	657.58	245	164.6	244.77
$\lambda =50$	2431.2	831.78	245	154	245.02
$\lambda =60$	2553	1069.18	245	161.97	244.9
$\lambda =70$	3564.2	1315.12	245	164.5	243.52
$\lambda =80$	3697.3	1583.12	245	165	243.68
$\lambda =90$	4194.4	1919.5	245	161.9	243.78

,

Table 11. Performance comparison of $P\_ztrue$ when ${\sigma }_v^{}=30$.

$\mathit{P}\_\mathit{z}\mathit{t}\mathit{r}\mathit{u}\mathit{e}$	JPDA	MHT	FCM	GLMB	TDA
$\lambda =0$	1	1	0.9671	0.4468	0.9992
$\lambda =10$	0.3373	0.7736	0.9683	0.3807	0.9957
$\lambda =20$	0.2068	0.5986	0.9605	0.3634	0.9941
$\lambda =30$	0.1471	0.4670	0.9613	0.3470	0.9901
$\lambda =40$	0.1147	0.3558	0.9538	0.3269	0.9865
$\lambda =50$	0.0946	0.2820	0.9533	0.3280	0.9830
$\lambda =60$	0.0803	0.2202	0.9500	0.3607	0.9800
$\lambda =70$	0.0688	0.1789	0.9442	0.3695	0.9826
$\lambda =80$	0.0611	0.1488	0.9429	0.3487	0.9818
$\lambda =90$	0.0544	0.1227	0.9424	0.3276	0.9777

,

Table 12. Performance comparison of $P\_equal$ when ${\sigma }_v^{}=30$.

$\mathit{P}\_\mathit{e}\mathit{q}\mathit{u}\mathit{a}\mathit{l}$	JPDA	MHT	FCM	GLMB	TDA
$\lambda =0$	0	1	0.6411	0.033	0.9164
$\lambda =10$	0	0.8012	0.6376	0.0252	0.9149
$\lambda =20$	0	0.6301	0.6381	0.0231	0.9159
$\lambda =30$	0	0.4986	0.6360	0.0216	0.9218
$\lambda =40$	0	0.3838	0.6375	0.0213	0.9113
$\lambda =50$	0	0.3051	0.6290	0.0198	0.9070
$\lambda =60$	0	0.2404	0.6275	0.0234	0.9044
$\lambda =70$	0	0.1965	0.6293	0.0248	0.8775
$\lambda =80$	0	0.1637	0.6270	0.0229	0.8749
$\lambda =90$	0	0.1368	0.6238	0.0229	0.8871

and

Table 13. Performance comparison of $\mathit{n}\mathit{u}\mathit{m}\_\mathit{o}\mathit{b}\mathit{s}$ when ${\sigma }_v^{}=30$.

$\mathit{n}\mathit{u}\mathit{m}\_\mathit{o}\mathit{b}\mathit{s}$	JPDA	MHT	FCM	GLMB	TDA
$\lambda =0$	5	5	5	51.1	5.03
$\lambda =10$	5	7.28	5	51	5.01
$\lambda =20$	5	7.86	5	51.1	5.02
$\lambda =30$	5	8.18	5	50.5	5
$\lambda =40$	5	8.54	5	51.1	5
$\lambda =50$	5	8.68	5	50.9	5
$\lambda =60$	5	8.7	5	50.93	5.01
$\lambda =70$	5	8.28	5	51	5.23
$\lambda =80$	5	8.14	5	51	5.24
$\lambda =90$	5	8.06	5	51.4	5.13

show the performance comparison of five algorithms when ${\sigma }_v^{}=30,\lambda =0,10,20,\dots ,90$. Table 8 and Table 9 show the performance comparison of $num\_all$ and $P\_all$, respectively. As can be seen from the table, the $P\_all$ of TDA, MHT, and FCM is relatively stable, and the $P\_all$ of TDA is the highest.

Table 10 and Table 11 show the performance comparison of $num\_zall$ and $P\_ztrue$, respectively. As shown in the table, the $P\_ztrue$ of TDA and FCM is high and stable. FCM is based on the principle of “one target corresponds to one measurement”, and the number of targets is assumed to be known, so $P\_ztrue$ is higher. MHT adopts the principle of “target: measurement = 1: N”. JPDA is greatly affected by clutter. $num\_zall$ increases as clutter density increases, and $P\_ztrue$ decreases for both MHT and JPDA. GLMB is slightly affected by clutter, and its performance is stable. However, there are so many cases of incorrect association in the association process that make $P\_ztrue$ of GLMB very low.

Table 12 and Table 13 show the performance comparison of $P\_equal$ and $num\_obs$, respectively. As can be seen from the table, JPDA and FCM assume that the number of targets is known; $num\_obs$ of TDA is the most accurate, followed by MHT. Compared with the $num\_obs$ of other algorithms, the difference of GLMB is larger, and there are clearly too many interruptions of association. Combined with $P\_all$ and $P\_ztrue$, the performance difference between FCM and TDA is not significant. However, according to $P\_equal$, the association performance of TDA is better. Obviously, when the targets do parallel motion or hedging motion, there are track crossings and track blending in the association results of FCM. The $P\_equal$ of JPDA is determined by $num\_all$, $num\_zall$, $num\_obs$, and other factors. The $num\_obs$ is known, and the $num\_zall$ is significantly higher than other algorithms, which makes the $P\_equal$ small. So the $P\_equal$ are all 0.

In summary, when ${\sigma }_v^{}=30$, the association performance of TDA is the best, followed by FCM, MHT, JPDA, and GLMB. When solving practical problems, the algorithm complexity is high, and the algorithm performance is not necessarily good. The complexity of GLMB is obviously higher than other algorithms, but its association results are not satisfactory. JPDA and MHT are affected by clutter, and their performance is unstable. FCM is difficult to deal with strong maneuvering targets effectively. Especially, when multiple targets perform special maneuvers, the performance of FCM algorithm will be affected. Moreover, FCM, MHT, JPDA, and GLMB require many assumptions, such as target motion model, the number of targets, and other information that is impossible to know when solving the practical problem. Compared with FCM, MHT, JPDA, and GLMB, TDA does not require assuming any unpredictable information as the priori information. It can also utilize the learning network to process the data quickly and efficiently and reduce the complexity of algorithm. Moreover, TDA provides solutions to various problems that may exist in the process of multi-target data association and ensures the accuracy of association results. Based on RL technology, TDA creates a new multi-target data association architecture, which breaks all the limitations of traditional data association algorithms and significantly improves the association accuracy. It has certain practical application value.

8. Discussion

In order to solve the problems that may occur in the unknown system model, such as strong clutter interference, sensor detection error, target strong maneuvering, and new target suddenly appearing, this paper proposes a multi-target data association algorithm based on RL technology. This algorithm abandons the traditional data association framework and combines with RL technology to build a new multi-target data association network architecture. It can not only combine the dynamic exploration ability of RL and the long-term memory function of LSTM network to predict the association probability between measurement and target but also use the Bayesian network and multi-order least squares curve fitting method to predict the target position. After that, the output value is fed into the Bayesian recursive function to calculate the reward. In addition, some mechanisms to solve the practical problems are established for the characteristics of target and measurement data. Finally, it realizes the accurate association between the point and the track. The performance of this algorithm is verified by comparing with four classical and advanced algorithms in different environments. Moreover, quite complex scenarios are designed in the test process of multi-target data association. Both multiple targets in the form of formation for crossover, parallel, and other complex movement modes, as well as strong maneuver models such as turning and acceleration during the target movement were added. The special situation that new targets may suddenly appear during the association process is also fully taken into account. Such complex scenarios cover almost all possible situations in real scenarios. The experimental results demonstrate that this algorithm can still maintain a good association performance in the face of large-scale target scenarios.

This paper breaks through the shackles of traditional data association and innovatively puts forward a new idea to solve the problem of multi-target data association. When the real scenario has dense clutter distribution, strong target mobility, and large sensor errors, it is still possible to achieve accurate association between points and tracks of multiple targets, which reflects the engineering application value of this algorithm and has high theoretical research significance. Likewise, this algorithm has certain limitations. When there is stealthy target movement or a large amount of electronic interference in the environment, the points originating from the target in the measurement data acquired by the sensor may be missing, and the association is easily broken. This algorithm faces great challenges, and its performance cannot be guaranteed. During the simulation experiments, as ${\sigma }_v^{}$ and $\lambda$ become larger, mechanisms such as Agent detection and Ring Wave Gate Screening consume more and more time. Moreover, the frequent false association phenomenon causes the number triggering the Adaptive Adjustment Mechanism to increase and take up more and more time. The average running time of a single simulation experiment has been tested to increase by a factor of about 10, and the computational complexity is quite high. Therefore, the focus of subsequent research is to break the limitations of this algorithm and improve the performance and practicality of algorithm.