Gaussian Mixture Probability Hypothesis Density Filter for Heterogeneous Multi-Sensor Registration

Abstract

Spatial registration is a prerequisite for data fusion. Existing methods primarily focus on similar sensor scenarios and rely on accurate data association assumptions. To address the heterogeneous sensor registration in complex data association scenarios, this paper proposes a Gaussian mixture probability hypothesis density (GM-PHD)-based algorithm for heterogeneous sensor bias registration, accompanied by an adaptive measurement iterative update algorithm. Firstly, by constructing augmented target state motion and measurement models, a closed-form expression for prediction is derived based on Gaussian mixture (GM). In the subsequent update, a two-level Kalman filter is used to achieve an approximate decoupled estimation of the target state and measurement bias, taking into account the coupling between them through pseudo-likelihood. Notably, for heterogeneous sensors that cannot directly use sequential update techniques, sequential updates are first performed on sensors that can obtain complete measurements, followed by filtering updates using extended Kalman filter (EKF) sequential update techniques for incomplete measurements. When there are differences in sensor quality, the GM-PHD fusion filter based on measurement iteration update is sequence-sensitive. Therefore, the optimal subpattern assignment (OSPA) metric is used to optimize the fusion order and enhance registration performance. The proposed algorithms extend the multi-target information-based spatial registration algorithm to heterogeneous sensor scenarios and address the impact of different sensor-filtering orders on registration performance. Our proposed algorithms significantly improve the accuracy of bias estimation compared to the registration algorithm based on significant targets. Under different detection probabilities and clutter intensities, the average root mean square error (RMSE) of distance and angular biases decreased by 11.8% and 8.6%, respectively.

1. Introduction

In sensor networks, information gathered from various sensors is collaboratively fused to enhance the overall system performance [1]. Sensor fusion techniques have found extensive applications in fields such as autonomous mobile systems [2], remote sensing [3], wireless sensor networks [4], and target tracking [5], benefiting from the rapid development of sensor technology. Spatial registration is a critical aspect of data fusion in multi-sensor systems. As shown in Figure 1, it is essential to estimate and correct the measurement biases (such as angle and distance) of the sensors. This allows the measurements from multiple sensors to be processed in a common reference system. Failure to perform spatial registration and directly fuse sensor data can lead to degraded tracking results, target loss, and even inferior performance compared to single-sensor processing.

Currently, most information fusion systems treat spatial registration and track association as separate steps. Initially, significant target track association is performed, followed by spatial registration based on a single significant target. These methods can yield good bias registration results when significant targets are correctly extracted. However, these methods have limitations. In multi-sensor multi-target tracking scenarios, spatial registration and track association are interconnected, and the presence of sensor biases presents significant challenges to track association [6,7,8,9,10].

In recent years, the application of finite set statistics (FISST) theory has emerged as a solution to address multi-target tracking problems without the need for data association [11,12]. Within the FISST framework, target states and measurements are modeled as random finite sets (RFS), allowing for the tracking of an unknown and time-varying number of targets in cluttered environments. By constructing multi-target transition densities and multi-target likelihood functions, multi-target tracking can be effectively described within a rigorous Bayesian framework. However, the optimal implementation of multi-target Bayesian filters becomes challenging due to the combinatorial nature of multiple integrals and multi-target densities. To overcome this challenge, Mahler introduced the probability hypothesis density (PHD) filter as a first-order statistical moment approximation of the multi-target posterior density [13].

There are two primary methods for implementing PHD recursion: sequential Monte Carlo (SMC) and Gaussian mixture (GM). The SMC-PHD filter employs a large number of particles to approximate multidimensional integrals, resulting in a high computational burden [14,15]. Furthermore, clustering techniques are necessary to extract target state estimates. To address these limitations, Vo et al. introduced the GM-PHD filter, which analytically propagates weights, means, and covariances using Kalman filtering [16,17,18,19]. For multi-sensor multi-target tracking, Mahler proposed the iterated corrector PHD (IC-PHD) filter [20]. This filter follows an iterative approach, sequentially processing the filtering results of the previous sensor using the subsequent sensor. While the concept of this filter is straightforward, its tracking performance can be influenced by the order in which the sensors are processed [21,22]. In order to perform spatial registration alongside filtering, Lian et al. integrated spatial registration into the GM-PHD framework, enabling simultaneous filtering and registration. However, their method only addressed spatial registration for sensors of the same type [9,23,24,25].

All existing spatial registration methods for heterogeneous sensors in the current literature depend on significant targets (obtained through additional association relationships). Therefore, this study focuses on the spatial registration of heterogeneous multi-sensor systems within the RFS framework. In the RFS framework, there is no need for additional association relationships to extract common significant targets, and complete information from multiple targets can be utilized for spatial registration. While a previous study has developed a similar sensor spatial registration algorithm based on multi-target information within the RFS framework, the main objective of this paper is to extend the spatial registration algorithms proposed in [9,23,24,25] to a wider range of heterogeneous sensor scenarios. However, this extension presents three key challenges:

(1)

The nonlinear relationship between the state space and measurement space in recursive filtering of heterogeneous sensors causes coupling between target states and measurement biases.

(2)

Heterogeneous sensors yield data with varying dimensions for target detection (including complete and incomplete measurements), posing challenges for multi-sensor sequential filtering.

(3)

The final fusion and registration results can be affected by the varying sequential filtering order due to performance differences among sensors.

To address these problems, two algorithms are proposed: the heterogeneous sensor-augmented state registration based on GM-PHD random fusion order (HSAR-GM-PHD-R) and the heterogeneous sensor-augmented state registration based on GM-PHD adaptive observation with iterative updates (HSAR-GM-PHD-AO). It is important to note that this paper introduces two heterogeneous sensor spatial registration algorithms. The HSAR-GM-PHD-R method addresses the first two technologies, while the HSAR-GM-PHD-AO method tackles all three technical challenges.

The contributions are as follows:

(1)

By constructing augmented target motion and measurement models and using PHD recursion, a closed-form expression for augmented state prediction is derived based on Gaussian mixture models. Additionally, a two-level Kalman filter is used in the update to approximately decouple the estimation of target state and measurement biases.

(2)

For the registration problem of heterogeneous sensors, measurements obtained from the sensors are divided into complete measurements and incomplete measurements. Sequential updates are performed first for sensors that provide complete measurements, followed by filtering updates for incomplete measurements using EKF sequential update techniques.

(3)

To address the sensitivity of the sequence fusion-filtering algorithm to sensor quality differences, real-time evaluation of fusion consistency for each sensor is performed using the optimal subpattern assignment (OSPA) metric when the data quality of each sensor is unknown. By optimizing the fusion order, more accurate fusion results can be obtained.

This paper is organized as follows. Section 2 presents related work on multi-sensor spatial registration. The problem formulation is introduced in Section 3. Section 4 demonstrates the proposed spatial registration methods. The simulation results for four different scenarios are given in Section 5. Conclusions are in Section 6.

2. Related Work

Sensor spatial registration has applications in robotics, augmented reality, target tracking, navigation, autonomous vehicles, remote sensing, and industrial automation. Extensive research has been conducted on spatial registration for multi-sensor systems [23,24,25,26,27,28,29,30,31,32,33,34,35]. Spatial registration can be broadly divided into two categories: one is based on common significant targets, which rely on known association relationships, and the other is based on the RFS framework. The spatial registration based on RFS does not require the extraction of common significant targets but can directly use multi-target information for registration, thus avoiding the impact of data association on spatial registration. In current research, the RFS-based spatial registration method has shown superior performance compared to the common significant target-based approach. Although our study focuses on the RFS framework, we will provide a detailed summary of both types of methods in the following paragraphs to ensure comprehensive analysis and comparison.

Spatial registration algorithms based on significant targets include least squares (LS) [26,27,28], general least squares (GLS) [29,30], real-time quality control (RTQC) [31], exact maximum likelihood (EML) [32,33], and maximum likelihood registration (MLR) [34,35]. Among these algorithms, RTQC and LS disregard the influence of sensor measurement noise, limiting their performance to scenarios with low measurement noise. While the GLS algorithm considers measurement noise, it can only estimate system biases pairwise, similar to the LS algorithm, which hinders the achievement of optimal performance. The EML algorithm utilizes measured values of sensors in the system plane and employs the maximum likelihood principle to estimate the target position and sensor biases simultaneously. It approximates the sine and cosine results of the system biases during the coordinate transformation, establishes a linearized model between biases and measurements, and iteratively obtains bias estimates. MLR improves upon this approximation method by establishing a linearized model between biases and measurements based on Taylor expansion, resulting in bias estimation results close to the Cramer–Rao lower bound. However, these algorithms assume well-established data association.

In complex target scenarios, where the collaboration of multiple types of sensors is required, the spatial registration of heterogeneous sensors becomes a significant research focus in the field of sensor fusion. Heterogeneous sensor measurements can be classified into two categories. The first category is complete measurements, which provide the full 3D position of the target obtained from sensors like radars. The second category is incomplete measurements or missing information measurements, which include one-dimensional or two-dimensional measurement information obtained from sensors such as optical cameras, infrared sensors, and electronic support measures [35]. For spatial registration involving sensors with complete measurements, the maximum likelihood registration algorithm can achieve estimation performance that closely approximates the Cramer–Rao lower bound. However, when dealing with incomplete measurements, this method can suffer from model mismatch, resulting in a decline in registration performance. To address this, reference [34] extended the maximum likelihood registration algorithm to the context of heterogeneous sensor registration by utilizing triangulation techniques to complete the missing measurement information, thereby enabling spatial registration for heterogeneous sensors. Nevertheless, it is important to note that the aforementioned methods for heterogeneous sensor spatial registration still rely on the presence of common significant targets. They essentially expand the approach based on common significant targets to accommodate the requirements of heterogeneous sensors.

The utilization of random finite set theory in solving multi-target tracking problems eliminates the need for data association. Therefore, conducting spatial registration within the RFS framework eliminates the need to select specific common significant targets and allows for the full utilization of multi-target detection information. Currently, many researchers have applied the PHD filter to multi-sensor spatial registration. Lian et al. proposed a spatial registration method based on the augmented state GM-PHD filter, utilizing Kalman filter techniques to estimate both target states and measurement biases [23]. However, the coupling between measurement biases and target states limits the effectiveness of this method in multi-target scenarios. Meanwhile, Ristic et al. relaxed the assumptions of nonlinear and non-Gaussian probability density functions and implemented spatial registration using sequential Monte Carlo particle filtering [36,37]. In order to enrich the measurement information, Wu et al. introduced a multi-target GM-PHD filter registration method based on Doppler information. This method incorporates Doppler information into the original measurements and performs joint estimation of target states and system biases by deriving a Gaussian mixture posterior density that incorporates Doppler measurement information [25]. Compared with the registration method proposed in reference [24], this approach demonstrates stronger robustness in dense clutter. It is important to note that these spatial registration methods proposed within the RFS framework only focus on spatial registration among sensors of the same type.

As the variety of sensor types and detection scenarios increases, there is a pressing need to investigate spatial registration algorithms for heterogeneous sensors. Therefore, the main focus of this paper is to extend the spatial registration algorithms within the RFS framework to heterogeneous sensor scenarios, while also addressing the key technologies encountered in the extension process.

3. Problem Formulation

In order to simultaneously estimate the target state and measurement bias within a unified filtering framework, references [23,24,25] combine them into an augmented state. In this section, we discuss the linear dynamic and measurement models for augmented target states within the GM-PHD filter framework. We first review the linear dynamic equations for the augmented state vector. Then, we introduce the linear pseudo-measurement equations based on the augmented target states. Finally, we present the random finite set formulation for multi-target filtering in linear Gaussian systems. This provides sufficient background information for introducing the registration method based on the augmented state GM-PHD filter.

3.1. Linear Gaussian Dynamical Model of the Augmented State

Let $x_{k,i}={\left[x_{k,i},{\dot{x}}_{k,i},y_{k,i},{\dot{y}}_{k,i},{\omega }_{k,i}\right]}^{\mathrm{T}}$ be the standard state of target $i$ at time $k$. Assuming a linear dynamic model for the target, i.e.,

$$x_{k,i}=F_{k-1}{x}_{k-1,i}+G{w}_{k-1}$$

(1)

where, assuming the process noise is zero-mean Gaussian white noise, $w_k~\mathcal{N}\left(0,Q_k\right)$. $Q_k=diag\left(\left[{\sigma }_{k,w}^2,{\sigma }_{k,w}^2,{\sigma }_{k,\omega }^2\right]\right)$ represents the covariance matrix of process noise, and the state transition matrix $F$ can be expressed as

$$F=\left[\begin{array}{ccccc}1& \frac{\sin \hspace{0.33em}\omega T}{\omega }& 0& \frac{\cos \hspace{0.33em}\omega T-1}{\omega }& 0\\ 0& \cos \hspace{0.33em}\omega T& 0& -\sin \hspace{0.33em}\omega T& 0\\ 0& \frac{1-\cos \hspace{0.33em}\omega T}{\omega }& 1& \frac{\sin \hspace{0.33em}\omega T}{\omega }& 0\\ 0& \sin \hspace{0.33em}\omega T& 0& \cos \hspace{0.33em}\omega T& 0\\ 0& 0& 0& 0& 1\end{array}\right]$$

(2)

where the discrete time interval is $T$, consistent with the sampling intervals of all sensors, and

$$G=\left[\begin{array}{ccc}\frac{T^2}{2}& 0& 0\\ T& 0& 0\\ 0& \frac{T^2}{2}& 0\\ 0& T& 0\\ 0& 0& T\end{array}\right]$$

(3)

Assuming there are $N_S$ unregistered sensors, and their measurement biases are mutually independent, the dynamics of each measurement bias can also be modeled as a first-order Gaussian Markov process.

$${\beta }_{k,s}={\beta }_{k-1,s}+n_{k-1,s},\hspace{1em}s=1,\dots ,N_S$$

(4)

where $n_{k-1,s}$ denotes the process noise of sensor $s$. The process noise is assumed to follow a zero-mean Gaussian white noise, i.e., $n_{k-1,s}~\mathcal{N}\left(0,B_{k-1,s}\right)$, where $B_{k-1,s}$ represents the covariance matrix of the process noise associated with measurement bias.

An augmented state vector ${\overline{x}}_{k,i}={\left[x_{k,i};{\beta }_{k,1};\dots ;{\beta }_{k,N_S}\right]}^T$ is constructed with its motion model as

$${\overline{x}}_{k,i}={\overline{F}}_{k-1}{\overline{x}}_{k-1,i}+{\overline{w}}_{k-1}$$

(5)

where ${\overline{F}}_{k-1}$ represents the augmented state transition matrix, i.e., ${\overline{F}}_{k-1}=\text{blk-diag}\left(F_{k-1},I_{n_1\times n_1},\dots ,I_{n_S\times n_S}\right)$, while $n=\dim \left({\beta }_{k,s}\right)$ denotes the dimensions of measurement bias. ${\overline{w}}_{k-1}~\mathcal{N}\left(0,{\overline{Q}}_{k-1}\right)$ represents the augmented process noise, with ${\overline{Q}}_{k-1}=\text{blk-diag}\left(Q_{k-1},B_{k-1,1},\dots ,B_{k-1,N_S}\right)$ being the augmented process covariance matrix and blk-diag representing a block-diagonal matrix.

3.2. Linear Gaussian Measurement Model of the Augmented State

Considering $N_S$ sensors performing measurements in the observation scenario, if the m-th measurement from sensor $s$ at time $k$ is associated with target $i$, the nonlinear measurement equation is given by

$$z_{k,m}^s=h\left(x_{k,i},x_{k,s}\right)+{\beta }_{k,s}+{\upsilon }_{k,s}$$

(6)

where $x_{k,i}$ represents the state vector of target $i$ at time $k$, and $x_{k,s}$ denotes the known position of sensor $s$. $h\left(x_{k,i},x_{k,s}\right)$ is a nonlinear mapping function from target $i$ in the common coordinate system to the measurement coordinate system of sensor $s$. ${\beta }_{k,s}$ represents the measurement bias of sensor $s$. ${\upsilon }_{k,s}$ represents zero-mean Gaussian white noise, denoted as ${\upsilon }_{k,s}~\mathcal{N}\left(0,R_{k,s}\right)$.

To linearize the measurement Equation (6), a first-order Taylor expansion is performed on $h\left(x_{k,i},x_{k,s}\right)$ around the predicted position ${\widehat{x}}_{k\left|k-1\right.,i}$, resulting in

$$\begin{array}{l}{z}_{k,m}^s=h\left(x_{k,i},x_{k,s}\right)+{\beta }_{k,s}+{\upsilon }_{k,s}\\ \approx h\left({\widehat{x}}_{k\left|k-1\right.,i},x_{k,s}\right)+H_{k,s}\left(x_{k,i}-{\widehat{x}}_{k\left|k-1\right.,i}\right)+{\beta }_{k,s}+{\upsilon }_{k,s}\\ =H_{k,s}{x}_{k,i}+h\left({\widehat{x}}_{k\left|k-1\right.,i},x_{k,s}\right)-H_{k,s}{\widehat{x}}_{k\left|k-1\right.,i}+{\beta }_{k,s}+{\upsilon }_{k,s}\end{array}$$

(7)

where $H_{k,s}$ represents the Jacobian matrix of the nonlinear measurement function $h\left(x_{k,i},x_{k,s}\right)$ of sensor $s$ at position ${\widehat{x}}_{k\left|k-1\right.,i}$. According to Equation (6), the pseudo-measurement $z_{c,k,m}^s$ is constructed as

$$z_{c,k,m}^s=z_{k,m}^s+H_{k,s}{\widehat{x}}_{k\left|k-1\right.,i}-h\left({\widehat{x}}_{k\left|k-1\right.,i},x_{k,s}\right)$$

(8)

If the measurement $z_{k,m}^s$ belongs to the target state $x_{k,i}$, the pseudo-measurement $z_{c,k,m}^s$ of $z_{k,m}^s$ is related to the true target state as

$$z_{c,k,m}^s=H_{k,s}{x}_{k,i}+{\beta }_{k,s}+{\upsilon }_{k,s}$$

(9)

Equation (9) is reformulated for the pseudo-measurement as

$$\begin{array}{ll}{z}_{c,k,m}^s& =\left[H_{k,s},{\Psi }_1,\dots ,{\Psi }_{N_S}\right]{\left[x_{k,i}^T,{\beta }_{k,1}^T,\dots ,{\beta }_{k,N_S}^T\right]}^T+{\upsilon }_{k,s}\\ & ={\overline{H}}_{k,s}{\overline{x}}_{k,i}+{\upsilon }_{k,s}\end{array}$$

(10)

where ${\overline{H}}_{k,s}$ is referred to as the augmented observation matrix, and ${\Psi }_r$ is written as

$${\Psi }_r=\left\{\begin{array}{cc}{I}_{n_r\times n_r}& r=s\\ 0& r\ne s\end{array}\right.$$

(11)

3.3. Random Finite Set Formulation of Multi-Target Filtering

The enhanced dynamic model for multiple moving targets and the multi-sensor measurement system model are integrated into the RFS framework. Let $N_{k-1}$ denote the number of targets at time $k-1$, with their states represented by ${\overline{x}}_{k-1,1},\dots ,{\overline{x}}_{k-1,N_{k-1}}$. At time $k$, these targets may either continue to exist or disappear, new targets may emerge, and existing targets may give rise to new ones. At the same time, consider that sensor $s$ has $M_k^s$ measurements, denoted as $z_{k,1}^s,\dots ,z_{k,M_k^s}^s$. These measurements can originate from targets or clutter. Based on RFS theory, the target states and measurements are represented as

$${\overline{X}}_k=\left\{{\overline{x}}_{k,1},\dots ,{\overline{x}}_{k,N_k}\right\}$$

(12)

$$Z_k^s=\left\{z_{k,1}^s,\dots ,z_{k,M_k^s}^s\right\}$$

(13)

For the standard multi-target motion model, given the multi-target state ${\overline{X}}_{k-1}$ at time $k-1$, the multi-target state set ${\overline{X}}_k$ at time $k$ is composed of three components: newborn targets, surviving targets, and derived targets, denoted as

$${\overline{X}}_k={\overline{\Gamma }}_k{\displaystyle \cup \left[{\displaystyle \underset{{\overline{x}}_{k-1}\in {\overline{X}}_{k-1}}{\cup }{S}_{k|k-1}\left({\overline{x}}_{k-1}\right)}\right]}{\displaystyle \cup \left[{\displaystyle \underset{{\overline{x}}_{k-1}\in {\overline{X}}_{k-1}}{\cup }{B}_{k|k-1}\left({\overline{x}}_{k-1}\right)}\right]}$$

(14)

where ${\overline{\Gamma }}_k$ represents the RFS of newborn target states at time $k$. The survival target RFS is denoted as $S_{k|k-1}\left({\overline{x}}_{k-1}\right)$, with a probability $p_{S,k}\left({\overline{x}}_{k-1}\right)$ to persist to the next moment, or with a probability $1-p_{S,k}\left({\overline{x}}_{k-1}\right)$ to disappear. $B_{k|k-1}\left({\overline{x}}_{k-1}\right)$ represents the RFS of target derivations based on the target state ${\overline{x}}_{k-1}$, which is not considered in this paper.

Based on this, the multi-target transition density from ${\overline{x}}_{k-1,i}$ to $\overline{x}$ can be expressed as

$$f_{k|k-1}\left(\overline{x}|{\overline{x}}_{k-1,i}\right)=\mathcal{N}\left(\overline{x};{\overline{F}}_{k-1}{\overline{x}}_{k-1,i},{\overline{Q}}_{k-1}\right)$$

(15)

Furthermore, it is assumed that the survival probability is independent of the target state, i.e.,

$$p_{S,k}\left({\overline{x}}_{k-1,i}\right)=p_{S,k}$$

(16)

Moreover, it is also assumed that the RFS involved in the union operation in Equation (14) are mutually independent.

Given the multi-target state RFS ${\overline{X}}_k$, the corresponding measurement RFS consists of two parts: clutter and target. The measurement RFS $Z_k^s$ is defined as

$$Z_k^s=K_k^s{\displaystyle \cup \left[{\displaystyle \underset{{\overline{x}}_{k,i}\in {\overline{X}}_k}{\cup }{\Theta }_k^s\left({\overline{x}}_{k,i}\right)}\right]}$$

(17)

where $K_k^s$ represents the RFS of clutter, and ${\Theta }_k^s\left({\overline{x}}_{k,i}\right)$ is the RFS of target measurements obtained from the detection of target ${\overline{x}}_k$. The target is detected with a probability $p_{D,k}^s\left({\overline{x}}_{k,i}\right)$, generating measurement $\left\{z_{k,m}^s\right\}$, and may be missed with a probability of $1-p_{D,k}^s\left({\overline{x}}_{k,i}\right)$, with its measurement represented as $\varnothing$. In a linear Gaussian multi-target tracking system, the probability density of the measurement likelihood function can be modeled as

$$g_k\left(z_{k,m}^s|{\overline{x}}_{k,i}\right)=\mathcal{N}\left(z_{c,k,m}^s;{\overline{H}}_{k,s}{\overline{x}}_{k,i},R_s\right)$$

(18)

Assuming that the detection probability is independent of the state, then

$$p_{D,k}^s\left({\overline{x}}_{k-1,i}\right)=p_{D,k}^s$$

(19)

4. Methods

In this section, we address three key challenges encountered when extending the spatial registration algorithm proposed in [9,23,24,25] to heterogeneous sensor scenarios. As shown in Figure 2, we provide solutions for each challenge. Firstly, to address the coupling problem between the target state and measurement bias, we propose a two-level Kalman filtering approximation for decoupling. Secondly, as different-dimensional target detection data generated by heterogeneous sensors pose a challenge for multi-sensor filtering, we utilize EKF sequential filtering techniques to effectively filter and fuse data with varying dimensions. Lastly, considering the fusion and registration results can be affected by the varying sequential filtering order due to performance differences among sensors, we utilize the OSPA metric to evaluate real-time fusion consistency across sensors. By optimizing the fusion sequence, more precise fusion results can be achieved.

4.1. Augmented State GM-PHD Registration Based on Two-Level Kalman Filter

To tackle the problem of coupling between the target state and measurement bias discussed in [23], we utilize a two-level Kalman filter. Please note that this subsection primarily addresses spatial registration for similar sensors. The subsequent subsection will extend this registration method to heterogeneous sensors.

It is also assumed that the sets involved in the union operation in Equation (17) are mutually independent. Based on the assumption of the independence between target states and sensor biases, the PHD of the target birth density RFS ${\overline{\Gamma }}_k$ can be represented in the form of a Gaussian mixture

$$\begin{array}{l}{\gamma }_k\left(\overline{x}\right)={\displaystyle \sum _{l=1}^{L_{\gamma ,k}}{w}_{\gamma ,k}^l\mathcal{N}\left(\overline{x};{\overline{m}}_{\gamma ,k}^l,{\overline{P}}_{\gamma ,k}^l\right)}\\ ={\displaystyle \sum _{l=1}^{L_{\gamma ,k}}{w}_{\gamma ,k}^l\mathcal{N}\left(x;m_{\gamma ,k}^l,P_{\gamma ,k}^l\right)}{\displaystyle \prod _{s=1}^{N_S}\mathcal{N}\left({\beta }_s;{\beta }_{\gamma ,k,s}^l,B_{\gamma ,k,s}^l\right)}\end{array}$$

(20)

where ${\overline{m}}_{\gamma ,k}^l={\left[m_{\gamma ,k}^l;{\beta }_{\gamma ,k,1}^l;\dots ;{\beta }_{\gamma ,k,N_S}^l\right]}^T$, ${\overline{P}}_{\gamma ,k}^l=\text{blk-diag}\left(P_{\gamma ,k}^l,B_{\gamma ,k,1}^l,\dots ,B_{\gamma ,k,N_S}^l\right)$, $L_{\gamma ,k}$ is the number of newborn Gaussian components. $w_{\gamma ,k}^l$ represents the weights of the newborn Gaussian components. $m_{\gamma ,k}^l$ stands for the mean state of the newborn Gaussian components. $P_{\gamma ,k}^l$ denotes the covariance of the newborn Gaussian components. ${\beta }_{\gamma ,k,s}^l$ represents the mean bias of the newborn Gaussian components. $B_{\gamma ,k,s}^l$ is the covariance of the bias for the newborn Gaussian components, and $\left(l=1,\cdots ,L_{\gamma ,k},s=1,\cdots ,N_S\right)$.

Prediction: Assuming that the $v_{k-1}\left(\overline{x}\right)$ at time $k-1$ follows a Gaussian mixture form,

$$\begin{array}{l}{v}_{k-1}\left(\overline{x}\right)={\displaystyle \sum _{l=1}^{L_{k-1}}{w}_{k-1}^l\mathcal{N}\left(\overline{x};{\overline{m}}_{k-1}^l,{\overline{P}}_{k-1}^l\right)}\\ ={\displaystyle \sum _{l=1}^{L_{k-1}}{w}_{k-1}^l\mathcal{N}\left(x;m_{k-1}^l,P_{k-1}^l\right)}{\displaystyle \prod _{s=1}^{N_S}\mathcal{N}\left({\beta }_s;{\beta }_{k-1,s}^l,B_{k-1,s}^l\right)}\end{array}$$

(21)

where ${\overline{m}}_k^l={\left[m_k^l;{\beta }_{k,1}^l;\dots ;{\beta }_{k,N_S}^l\right]}^T$, ${\overline{P}}_k^l=\text{blk-diag}\left(P_k^l,B_{k,1}^l,\dots ,B_{k,N_S}^l\right)$. $L_k$ is the number of Gaussian components. $w_k^l$ represents the weights of the Gaussian components. $m_k^l$ stands for the mean state of the Gaussian components. $P_k^l$ denotes the covariance of the state for the Gaussian components. ${\beta }_{k,s}^l$ represents the mean bias of the Gaussian components, and $B_{k,s}^l$ is the covariance of the bias for the Gaussian components.

The prediction density $v_{k|k-1}$ at time $k$ also takes the form of a Gaussian mixture, given by

$$\begin{array}{ll}{v}_{k|k-1}\left(\overline{x}\right)& ={\gamma }_k\left(\overline{x}\right)+v_{S,k|k-1}\left(\overline{x}\right)\\ & ={\displaystyle \sum _{l=1}^{L_{k|k-1}}{w}_{k|k-1}^l\mathcal{N}\left(\overline{x};{\overline{m}}_{k|k-1}^l,{\overline{P}}_{k|k-1}^l\right)}\\ & ={\displaystyle \sum _{l=1}^{L_{k|k-1}}{w}_{k|k-1}^l\mathcal{N}\left(x;m_{k|k-1}^l,P_{k|k-1}^l\right)}{\displaystyle \prod _{s=1}^{N_S}\mathcal{N}\left({\beta }_j;{\beta }_{k|k-1,s}^l,B_{k|k-1,s}^l\right)}\end{array}$$

(22)

where $m_{k|k-1}^l$ represents the predicted target state at time $k$. ${\beta }_{k|k-1,s}^l$ stands for the predicted measurement bias for sensor $s$. $P_{k|k-1}^l$ is the covariance of the target state prediction. $B_{k|k-1,s}^l$ represents the covariance of the measurement bias prediction. ${\overline{m}}_{k|k-1}^l={\left[m_{k|k-1}^l;{\beta }_{k|k-1,1}^l;\dots ;{\beta }_{k|k-1,N_S}^l\right]}^T$ and ${\overline{P}}_{k|k-1}^l=\text{blk-diag}\left(P_{k|k-1}^l,B_{k|k-1,1}^l,\dots ,B_{k|k-1,N_S}^l\right)$ are the predicted augmented state vector and augmented covariance matrix, respectively. ${\gamma }_k\left(\overline{x}\right)$ represents the density of the birth target, and $v_{S,k|k-1}$ is the density of the surviving target. $v_{S,k|k-1}$ is given by the following equation:

$$v_{S,k|k-1}\left(\overline{x}\right)=p_{S,k}{\displaystyle \sum _{l=1}^{L_{k-1}}{w}_{k-1}^l\mathcal{N}\left(\overline{x};{\overline{m}}_{S,k|k-1}^l,{\overline{P}}_{S,k|k-1}^l\right)}$$

(23)

where

$${\overline{m}}_{S,k|k-1}^l={\left[m_{S,k|k-1}^l;{\beta }_{k|k-1,1}^l;\dots ;{\beta }_{k|k-1,N_S}^l\right]}^T={\overline{F}}_{k-1}{\overline{m}}_{k-1}^l$$

(24)

$$\begin{array}{c}{\overline{P}}_{S,k|k-1}^l=\text{blk-diag}\left(P_{S,k|k-1}^l,B_{k|k-1,1}^l,\dots ,B_{k|k-1,N_S}^l\right)\\ ={\overline{F}}_{k-1}{\overline{P}}_{k-1}^l{\overline{F}}_{k-1}^T+{\overline{Q}}_{k-1}\end{array}$$

(25)

In Equations (24) and (25), $m_{S,k|k-1}^l$ and $P_{S,k|k-1}^l$ represent the predicted mean and predicted covariance of the surviving components, respectively. ${\beta }_{k|k-1,s}^l$ and $B_{k|k-1,j,l}$ represent the predicted mean and covariance of the bias components, respectively.

Expanding Equations (24) and (25), we obtain

$$\left\{\begin{array}{l}{m}_{S,k|k-1}^l=F_{k-1}{m}_{k-1}^l\\ P_{S,k|k-1}^l=F_{k-1}{P}_{k-1}^l{F}_{k-1}^T+Q_{k-1}\end{array}\right.$$

(26)

$$\left\{\begin{array}{l}{\beta }_{k|k-1,s}^l={\beta }_{k-1,s}^l\\ B_{k|k-1,s}^l=B_{k-1,s}^l+B_{k-1,s}\end{array}\right.$$

(27)

Similarly, Equation (23) can be expanded as

$$v_{S,k|k-1}\left(\overline{x}\right)=p_{S,k}{\displaystyle \sum _{l=1}^{L_{k-1}}{w}_{k-1}^l\mathcal{N}\left(x;m_{S,k|k-1}^l,P_{S,k|k-1}^l\right)}{\displaystyle \prod _{s=1}^{N_S}\mathcal{N}\left({\beta }_s;{\beta }_{k|k-1,s}^l,B_{k|k-1,s}^l\right)}$$

(28)

By combining Equations (20), (22), and (28), we obtain

$$\begin{array}{ll}{v}_{k|k-1}\left(\overline{x}\right)& ={\displaystyle \sum _{l=1}^{L_{\gamma ,k}}{w}_{\gamma ,k}^l\mathcal{N}\left(x;m_{\gamma ,k}^l,P_{\gamma ,k}^l\right)}{\displaystyle \prod _{s=1}^{N_S}\mathcal{N}\left({\beta }_s;{\beta }_{\gamma ,k,s}^l,B_{\gamma ,k,s}^l\right)}\\ & +p_{S,k}{\displaystyle \sum _{l=1}^{L_{k-1}}{w}_{k-1}^l\mathcal{N}\left(x;m_{S,k|k-1}^l,P_{S,k|k-1}^l\right)}{\displaystyle \prod _{s=1}^{N_S}\mathcal{N}\left({\beta }_s;{\beta }_{k|k-1,s}^l,B_{k|k-1,s}^l\right)}\end{array}$$

(29)

Update: We assume a measurement set from $N_S$ sensors at time $k$, based on the sequential sensor update approach, where the GM-PHD filter update equations are sequentially applied on each sensor. For sensor $r\left(r=1,\cdots ,N_S\right)$, the posterior intensity $v_{k|k,r}\left(\overline{x}\right)$ at time $k$ is also in the form of a Gaussian mixture.

$$v_{k|k,r}\left(\overline{x}\right)={\displaystyle \sum _{l=1}^{L_{k\left|k,r-1\right.}}\left(1-p_{D,k}^r\right)}{v}_{k|k,r-1}+{\displaystyle \sum _{z\in Z_k^r}{v}_{D,k,r}\left(\overline{x};z\right)}$$

(30)

where $v_{k|k,r-1}\left(\overline{x}\right)$ is the posterior density from sensor $r-1$, and

$$v_{D,k,r}\left(\overline{x};z\right)={\displaystyle \sum _{l=1}^{L_{k|k,r-1}}{w}_{k|k,r}^l\mathcal{N}\left(\overline{x};{\overline{m}}_{k|k,r}^l, {\overline{P}}_{k|k,r}^l\right)}$$

(31)

where $L_{k|k,r-1}$ is the number of Gaussian components from previous sensors. $w_{k|k,r}^l$ is the weight of Gaussian component $l$ after the update by sensor $r$. ${\overline{m}}_{k|k,r}^l$ and ${\overline{P}}_{k|k,r}^l$ are the augmented mean and covariance of Gaussian component $l$ after the update by sensor $r$. Assuming that sensor measurement biases are independent of the target states, Equation (31) can be transformed into

$$\begin{array}{ll}{v}_{D,k,r}\left(\overline{x};z\right)& ={\displaystyle \sum _{l=1}^{L_{k|k,r-1}}{w}_{k|k,r}^l\mathcal{N}\left(\overline{x};{\overline{m}}_{k|k,r}^l, {\overline{P}}_{k|k,r}^l\right)}\\ & \approx {\displaystyle \sum _{l=1}^{L_{k|k,r-1}}{w}_{k|k,r}^l\mathcal{N}\left(x;m_{k|k,r}^l, P_{k|k,r}^l\right)}{\displaystyle \prod _{s=1}^{N_S}\mathcal{N}\left({\beta }_s;{\beta }_{k|k,r}^{s,l}, B_{k|k,r}^{s,l}\right)}\end{array}$$

(32)

where $m_{k|k,r}^l$ and $P_{k|k,r}^l$ represent the target state and state covariance of the Gaussian component $l$ after the update by sensor $r$, respectively. ${\beta }_{k|k,r}^{s,l}$ and $B_{k|k,r}^{s,l}$ represent the estimated measurement bias of sensor $s$ and its corresponding covariance in Gaussian component $l$ after the update by sensor $r$, respectively.

Based on the sequential iterative fusion rule, let $v_{k|k,0}\left(\overline{x};z\right)\triangleq v_{k|k-1}\left(\overline{x}\right)$; for any Gaussian component $l$, $m_{k|k,0}^l\triangleq m_{k|k-1}^l$, $P_{k|k,0}^l\triangleq P_{k|k-1}^l$, ${\beta }_{k|k,0}^{s,l}\triangleq {\beta }_{k|k-1}^{s,l}$, $B_{k|k,0}^{s,l}\triangleq B_{k|k-1}^{s,l}$, the estimates of state and bias values are updated through the pseudo-measurement $z_{c,k,m}^r,m=1,\dots ,\left|M_k^r\right|$.

$${\overline{m}}_{k|k,r}^l={\overline{m}}_{k|k,r-1}^l+{\overline{G}}_{c,k,r}^l{\tilde{z}}_{c,k,m}^{r,l}$$

(33)

$${\overline{P}}_{k|k,r}^l=\left[I-{\overline{G}}_{c,k,r}^l{\overline{H}}_{k,r}^l\right]{\overline{P}}_{k|k,r-1}^l$$

(34)

where the augmented gain matrix ${\overline{G}}_{c,k,r}^l$ for Gaussian component $l$ corresponding to sensor $r$ can be expressed as

$${\overline{G}}_{c,k,r}^l=\left[G_{c,k,r}^l;K_{1,k,r}^l;\dots ;K_{N_S,k,r}^l\right]={\overline{P}}_{k|k,r-1}^l{\overline{H}}_{k,r}^l{\left[S_{c,k,r}^l\right]}^{-1}$$

(35)

where the Jacobian matrix ${\overline{H}}_{k,r}^l$ is given by

$${\overline{H}}_{k,r}^l={\left.{\overline{H}}_{k,r}\right|}_{{\overline{m}}_{k|k,r-1}^l}$$

(36)

In the fusion Equations (33) and (34), ${\overline{m}}_{k|k,r-1}^l={\left[m_{k|k,r-1}^l;{\beta }_{k|k,r-1}^{1,l};\dots ;{\beta }_{k|k,r-1}^{N_S,l}\right]}^T$, ${\overline{P}}_{k|k,r-1}^l=\text{blk-diag}\left(P_{k|k,r-1}^l,B_{k|k,r-1}^{1,l},\dots ,B_{k|k,r-1}^{N_S,l}\right)$. To reduce computational complexity, ${\overline{m}}_{k|k,r-1}^l$ is block-diagonalized, and according to Equation (34), we have

$${\overline{P}}_{k|k,r}^l\approx \text{blk-diag}\left(P_{k|k,r}^l,B_{k|k,r}^{1,l},\dots ,B_{k|k,r}^{N_S,l}\right)$$

(37)

The detailed derivation of ${\overline{P}}_{k|k,r}^l$, $G_{c,k,r}^l$, $H_{k,r}^l$, $K_{s,k,r}^l$, and $S_{c,k,r}^l$ can be found in Appendix A.

Using the principle of block matrix decomposition, the expanded state is decomposed into the target state and individual sensor measurement bias vectors. The target state is then updated using the residual of the pseudo-measurements. The pseudo-measurement residuals are computed based on

$$\begin{array}{ll}{\tilde{z}}_{c,k,m}^{r,l}& =z_{c,k,m}^r-{\overline{H}}_{k,r}^l{\overline{m}}_{k|k,r-1}^l\\ & =z_{c,k,m}^r-H_{k,r}^l{m}_{k|k,r-1}^l-{\displaystyle \sum _{s=1}^{N_S}{\Psi }_s{\beta }_{k|k,r-1}^{s,l}}\\ & =z_{c,k,m}^r-H_{k,r}^l{m}_{k|k,r-1}^l-{\beta }_{k|k,r-1}^{s,l}\end{array}$$

(38)

After obtaining the residuals, sequential updates are performed on the target state, target state estimation error covariance, sensor measurement biases, and sensor measurement bias estimation error covariance, respectively.

$$m_{k|k,r}^l=m_{k|k,r-1}^l+G_{c,k,r}^l{\tilde{z}}_{c,k,m}^{r,l}$$

(39)

$${\overline{P}}_{k|k,r}^l=\left[I-{\overline{G}}_{c,k,r}^l{\overline{H}}_{k,r}^l\right]{\overline{P}}_{k|k,r-1}^l$$

(40)

$${\beta }_{k|k,r}^{s,l}={\beta }_{k|k,r-1}^{s,l}+K_{s,k,r}^l{\tilde{z}}_{c,k,m}^{r,l},\hspace{0.33em}s=1,\dots ,N_S$$

(41)

$$B_{k|k,r}^{s,l}=\left[I-K_{s,k,r}^l{\Psi }_s\right]B_{k|k,r-1}^{s,l},\hspace{0.33em}s=1,\dots ,N_S$$

(42)

Once the updated mean and variance are obtained, the corresponding Gaussian component weights are updated as

$$w_{k|k,r}^l=\frac{p_{D,k}^r{w}_{k|k,r-1}^l\mathcal{N}\left(z_{c,k,m}^r;H_{k,r}^l{m}_{k|k,r-1}^l+{\beta }_{k|k,r-1}^{s,l},S_{c,k,r}^l\right)}{{\kappa }_k^l+p_{D,k}^r{\displaystyle \sum _{l=1}^{L_{k|k,r-1}}{w}_{k|k,r-1}^l\mathcal{N}\left(z_{c,k,m}^r;H_{k,r}^l{m}_{k|k,r-1}^l+{\beta }_{k|k,r-1}^{s,l},S_{c,k,r}^l\right)}}$$

(43)

After each sensor update as described above, computational complexity can be reduced by performing the standard merging and pruning operations of the GM-PHD filter. These operations can be applied either after each sensor update or after all sensor fusion is completed. It is important to note that when the last sensor fusion is completed, its updated result should be considered as the posterior density after multi-sensor fusion, i.e.,

$$\begin{array}{ll}{v}_k\left(\overline{x}\right)& ={\displaystyle \sum _{l=1}^{L_k}{w}_k^l\mathcal{N}\left(\overline{x};{\overline{m}}_k^l,{\overline{P}}_k^l\right)}\\ & ={\displaystyle \sum _{l=1}^{L_k}{w}_k^l\mathcal{N}\left(x;m_k^l,P_k^l\right)}{\displaystyle \prod _{s=1}^{N_S}\mathcal{N}\left({\beta }_s;{\beta }_{k,s}^l,B_{k,s}^l\right)}\\ & =v_{k|k,N_S}\left(\overline{x}\right)\\ & ={\displaystyle \sum _{l=1}^{L_{k|k,N_S}}{w}_{k|k,N_S}^l\mathcal{N}\left(x;m_{k|k,N_S}^l,P_{k|k,N_S}^l\right)}{\displaystyle \prod _{s=1}^{N_S}\mathcal{N}\left({\beta }_s;{\beta }_{k|k,N_S}^{s,l},B_{k|k,N_S}^{s,l}\right)}\end{array}$$

(44)

By estimating the state, it is possible to obtain the number of scene targets and their estimated state values. Since this state estimation has already eliminated the influence of measurement biases, quantitative calculation of the measurement bias for each sensor can be achieved using the following equation:

$${\beta }_{k,s}=\frac{1}{c}{\displaystyle \sum _{l=1}^{L_k}{w}_k^l{\beta }_{k,s}^l},\hspace{0.33em}s=1,\dots ,N_S$$

(45)

where $c$ is a normalization constant.

$$c={\displaystyle \sum _{l=1}^{L_k}{w}_k^l}$$

(46)

The pseudocode for augmented state GM-PHD registration based on a two-level Kalman filter is provided in Algorithm 1.

Algorithm 1 Augmented state GM-PHD registration based on two-level Kalman filter

Step 1. Prediction             The augmented state prediction is achieved through a two-level Kalman             recursion based on Equations (26) and (27). Step 2. Update             The augmented state update is achieved through a two-level Kalman             recursion based on Equations (39)–(42). Step 3. Calculation of the measurement bias             The measurement bias for each sensor can be achieved using Equations             (45) and (46).

4.2. Heterogeneous Multi-Sensor Sequential Filtering

Currently, there are two main fusion structures for heterogeneous sensor fusion: centralized and distributed fusion. Distributed fusion involves sending local track information to a fusion center to achieve the fusion of multi-target posterior densities from local track nodes. In contrast, centralized fusion sends the measurement information of local sensors to the fusion center. This fusion structure does not require separate processing of local sensor data but processes the measurement data centrally at the fusion center to complete global track updates. Compared to the distributed fusion structure, centralized fusion utilizes raw data from each sensor and requires larger communication bandwidth, but offers slightly higher fusion accuracy [38,39,40].

4.2.1. Centralized Fusion Filtering of Heterogeneous Sensors Based on EKF

Multi-sensor centralized fusion primarily relies on three filtering methods: parallel filtering, data compression filtering, and sequential filtering [41]. Among them, parallel filtering merges all the measurements reported by sensors at time $k$ into a global measurement by expanding the dimensions, and then it performs a one-time update on the target state. However, due to the data expansion, high-dimensional matrix multiplication and inversion operations are introduced, leading to increased computational complexity. Consequently, this approach is rarely used in multi-sensor multi-target fusion. Data compression filtering is mainly suitable for data fusion among homogeneous sensors. For data fusion between heterogeneous sensors, the mismatch in measurement noise covariances can significantly affect the fusion results. Sequential filtering treats the measurements provided by each sensor at time $k$ as independent new measurements and processes them sequentially. This approach does not require extending all measurements, thus avoiding a significant increase in computational complexity. Furthermore, it independently updates the target state based on the measurements from each sensor, mitigating the need to match measurement noise covariances between sensors. Therefore, sequential filtering becomes the primary approach for handling data fusion in scenarios involving heterogeneous sensors.

It should be noted that this paper categorizes heterogeneous measurements into complete and incomplete measurements. Complete measurements include the target’s full three-dimensional position information (distance, elevation angle, azimuth angle), while incomplete measurements typically provide only the target’s angle information.

The central idea of sequential filtering for heterogeneous sensor data fusion, as shown in Figure 3, is to first update the target state using complete measurements (such as active radar measurements). The filtered state estimate ${\overline{x}}_k^c$ obtained from the complete measurements is then used to replace the value of state estimation ${\overline{x}}_{k\left|k-1\right.}^{\mathrm{In}}$ for incomplete measurements (such as infrared measurements). Similarly, the filtered state estimation covariance ${\overline{P}}_k^c$ obtained from the complete measurements is used to replace the updated value of state estimation covariance ${\overline{P}}_{k\left|k-1\right.}^{\mathrm{In}}$ for incomplete measurement sensors. Subsequently, the target state is updated using the incomplete measurements, and the updated results serve as the global filtering state estimate ${\overline{x}}_k$ and state estimation covariance ${\overline{P}}_k$. The pseudocode for heterogeneous sensor registration using EKF sequential filtering is provided in Algorithm 2.

Algorithm 2. Heterogeneous sensor registration using EKF sequential filtering

INPUT: ${\overline{x}}_0$, ${\overline{P}}_0$, ${\left\{Z_k^{1:N_S}\right\}}_{k=1}^K$, the number of complete measurement sensors $L$     OUTPUT: ${\overline{x}}_k$, ${\overline{P}}_k$

step 1. Prediction                 Calculate the one-step prediction state ${\overline{x}}_{k\left|k-1\right.}^c$ and covariance ${\overline{P}}_{k\left|k-1\right.}^c$ based on (29).     step 2. Update the target state using complete measurements (e.g., active radar     measurements).                      step 3. Update the target state using incomplete measurements (e.g., infrared     measurements).                      step 4. Assign the filtering state estimate ${\overline{x}}_{k,N_S}^{\mathrm{In}}$ and covariance ${\overline{P}}_{k,N_S}^{\mathrm{In}}$ from incomplete measurements to the global filtering state estimate ${\overline{x}}_k$ and state estimate covariance ${\overline{P}}_k$

4.2.2. Recursive GM-PHD Filtering for Sequential Fusion with Heterogeneous Sensors

To address the multi-target filtering and registration in heterogeneous sensors within the RFS framework, this subsection combines the concepts of EKF filtering and sequential fusion [42]. By employing the GM-PHD filtering approach, it enables recursive operations on the states of multiple targets from heterogeneous sensors. The specific approach involves first updating the prior GM-PHD of multiple targets at time $k$ using complete measurements and using the updated results as the predicted values for the incomplete measurement sensors. Subsequently, in the fusion center, the incomplete measurements are utilized to update the target states and obtain the overall state estimation of multiple targets at time $k$.

• The prediction for the target state at time $k$ is shown in Equation (22).
• Let us denote $v_{k|k,1}\left(\overline{x}\right)=v_{k|k-1}\left(\overline{x}\right)$, then the equation for updating the target states with complete measurements at time $k$ can be expressed as

$$v_{k|k,r}^c\left(\overline{x}\right)={\displaystyle \sum _{l=1}^{L_{k\left|k,r-1\right.}}\left(1-p_{D,k}^{c,r}\right)}{v}_{k|k,r-1}^c+{\displaystyle \sum _{z\in Z_k^r}{v}_{D,k,r}^c\left(\overline{x};z\right)}$$

(47)

where $v_{k|k,r}^c$ ($r=1\cdots L$) represents the posterior GM-PHD at time $k$ after updating the states using complete measurements. $p_{D,k}^{c,r}$ is the detection probability of the complete measurement sensor, and ${\displaystyle \sum _{z\in Z_k^r}{v}_{D,k,r}^c\left(\overline{x};z\right)}$ denotes the contribution of the complete measurements to the target state update.

The process involves taking the updated results of the target states with complete measurements as predictions for incomplete measurements. Subsequently, the target states are updated using the incomplete measurements, and the resulting updated states with incomplete measurements are considered the global posterior GM-PHD obtained by the fusion system at time $k$.

$$\left\{\begin{array}{l}{v}_{k|k,r}^{\mathrm{In}}\left(\overline{x}\right)={\displaystyle \sum _{l=1}^{L_{k\left|k,r-1\right.}}\left(1-p_{D,k}^{\mathrm{In},r}\right)}{v}_{k|k,r-1}^{\mathrm{c}}+{\displaystyle \sum _{z\in Z_k^r}{v}_{D,k,r}^{\mathrm{In}}\left(\overline{x};z\right)}\\ v_{k|k}\left(\overline{x}\right)=v_{k|k,N_S}^{\mathrm{In}}\left(\overline{x}\right)\end{array}\right.$$

(48)

where $v_{k|k,r}^{\mathrm{In}}\left(\overline{x}\right)$ ($r=L+1\cdots N_S$) denotes the posterior GM-PHD of multiple targets at time $k$ after updating the target states using incomplete measurements. $p_{D,k}^{\mathrm{In},r}$ denotes the detection probability of the incomplete measurement sensor. ${\displaystyle \sum _{z\in Z_k^r}{v}_{D,k,r}^{\mathrm{In}}\left(\overline{x};z\right)}$ signifies the contribution of the incomplete measurements to the update of the target state. $v_{k|k}\left(\overline{x}\right)$ stands for the posterior GM-PHD of the fusion system at time $k$, obtained after a global update of the target states.

4.3. Heterogeneous Sensor Adaptive Measurement Iterative Update

In multi-sensor sequential filtering, different updating orders of sensors can lead to different tracking results. For instance, a tracking system consists of “good” sensors, denoted as $s-1$ sensors, and one “bad” sensor with low detection probability, high clutter intensity, and measurement noise. If the “bad” sensor is the $i$-th sensor, its negative impact may potentially be mitigated by the subsequent $s-i$ sensors in the sequential updating process. If the “bad” sensor is the last to update, the tracking results may suffer from false alarms, missed detections, and inaccurate state estimates, as there are no sensors to correct the negative influence caused by the “bad” sensor. In summary, the larger the value of $i$ for the “bad” sensor, the greater its negative impact on the tracking results of the system [22].

The GM-PHD fusion filtering algorithm based on sequential iterative update is sensitive to the update sequence, thus requiring the investigation of an adaptive optimization algorithm for fusion order. In practical applications, there is usually no prior information available about the quality of sensor measurements. Therefore, to optimize the observation sequence of sensors, online evaluation of the quality of multiple sensor data is necessary.

4.3.1. Measurement Consistency Metrics

To overcome the sensitivity of measurement sequential iteration results to fusion order when the data quality of each sensor’s measurements is unknown, a real-time assessment of measurement consistency for each sensor is performed using the OSPA distance. By optimizing the fusion order, more accurate fusion results are obtained. By selecting the state fusion results and the measurement sets from each sensor as finite sets at time $k-1$, the quality of sensor measurement data is assessed by calculating the OSPA distance between each finite set and the state fusion results.

We assume there are $N_S$ sensors, and the measurement set for sensor $s$ is denoted as ${\left\{z_{k-1,m}^s\right\}}_{m=1}^{M_{k-1}^s}$, and the state fusion result set is denoted as ${\left\{w_{k-1}^l,{\overline{m}}_{k-1}^l,{\overline{P}}_{k-1}^l\right\}}_{l=1}^{L_{k-1}}$. The OSPA distance for sensor $s$ with respect to the target state fusion set is defined as

$$D_k^s(z)={\left(\frac{1}{L_{k-1}}\left(\underset{\pi (s)\in \left\{1,2,\dots M_{k-1}^s\right\}}{\min }\sum _{l=1}^{L_{k-1}}{\Vert g\left({\overline{m}}_{k-1}^l\right),z_{k-1,\pi (s)}^s\Vert }^p+c^p\left|L_{k-1}-M_{k-1}^s\right|\right)\right)}^{1/p}$$

(49)

where $g\left({\overline{m}}_{k-1}^l\right)$ is obtained from Equation (6)

$$g\left({\overline{m}}_{k-1}^l\right)=h\left({\overline{m}}_{k-1}^l,x_{k-1,s}\right)$$

(50)

where $c$ represents the association sensitivity parameter, indicating the minimum truncation distance, and $p$ is the distance sensitivity parameter. Both parameters provide a comprehensive evaluation of the quality of target state and number. The measurement consistency metric reflects the consistency between the measurement set and the state fusion results. A smaller OSPA distance between sensor measurements and state fusion results indicates higher data quality for the sensor.

Based on Equations (49) and (50), the OSPA distance between sensor measurement data and state fusion results can be computed. These evaluation metrics are then sorted to determine the optimized sequence for the measurement sequential update at time $k$, resulting in more accurate fusion results.

4.3.2. Iterative Update Based on Consistency Metrics

When evaluating the consistency of heterogeneous sensors, the approach differs slightly from that of homogeneous sensors due to differences in data dimensions. In Figure 4, the first step involves measuring the consistency between the complete measurement and the state fusion results. The resulting measurements are then sorted in descending order. Next, the consistency between the incomplete measurement and the state fusion results is assessed. The measurements are ranked in descending order and appended to the sorted results of the complete measurement. This process generates the final consistency measurement results for heterogeneous sensor measurements. It is worth noting that, in this paper, the default incomplete sensors primarily include infrared and optical sensors, which exhibit higher angular measurement accuracy compared to complete measurement sensors, such as active 3D radar.

5. Experimental Results

In the domain of multi-sensor target tracking and data fusion, it is customary to select generic simulated scenarios (i.e., simulated scenarios provided in references [14,15,16,23,24,25]). However, our simulations extend these scenarios to heterogeneous sensor settings. Moreover, in complex detection scenarios, the main factors affecting target tracking and sensor registration are the sensor’s detection probability and the target’s clutter rate.

In this section, to evaluate our algorithm’s performance under different detection probabilities and clutter intensities, we conducted performance analysis using four different combinations based on ablation studies. OSPA distance and RMSE are widely used performance evaluation metrics in multi-target tracking, and we primarily employed them to assess the effectiveness of our proposed algorithms (HSAR-GM-PHD-R and HSAR-GM-PHD-AO). Lastly, we compared the HSAR-GM-PHD-AO algorithm with a heterogenous sensor registration algorithm based on significant targets to highlight its efficacy [43].

5.1. Simulation Scenarios

As shown in Figure 5, in a two-dimensional simulation scenario denoted as $\left[-1500, 1500\right]\times \left[0, 2000\right] {\mathrm{m}}^2$, there are four targets, all following a coordinated turn motion model. The target state is $x_{k,i}={\left[x_{k,i},{\dot{x}}_{k,i},y_{k,i},{\dot{y}}_{k,i},{\omega }_{k,i}\right]}^T$, including the position, velocity and angular velocity. The motion model for the target can be represented as

$$x_{k,i}=F_{k-1}{x}_{k-1,i}+G{w}_{k-1}$$

(51)

where the state transition matrix $F_{k-1}$ is given by

$$F_{k-1}=\left[\begin{array}{ccccc}1& \frac{\sin \hspace{0.33em}\omega T}{\omega }& 0& \frac{\cos \hspace{0.33em}\omega T-1}{\omega }& 0\\ 0& \cos \hspace{0.33em}\omega T& 0& -\sin \hspace{0.33em}\omega T& 0\\ 0& \frac{1-\cos \hspace{0.33em}\omega T}{\omega }& 1& \frac{\sin \hspace{0.33em}\omega T}{\omega }& 0\\ 0& \sin \hspace{0.33em}\omega T& 0& \cos \hspace{0.33em}\omega T& 0\\ 0& 0& 0& 0& 1\end{array}\right]$$

(52)

where all sensors have a sampling interval T = 1 s. The process noise covariance matrix $Q$ is given by

$$Q=\left[\begin{array}{ccc}{\sigma }_v^2\times \raisebox{1ex}{$T^2$}\!\left/ \!\raisebox{-1ex}{$2$}\right.& 0& 0\\ {\sigma }_v^2\times T& 0& 0\\ 0& {\sigma }_v^2\times \raisebox{1ex}{$T^2$}\!\left/ \!\raisebox{-1ex}{$2$}\right.& 0\\ 0& {\sigma }_v^2\times T& 0\\ 0& 0& {\sigma }_{\omega }^2\end{array}\right]$$

(53)

where ${\sigma }_v^2=5 \mathrm{m}/{\mathrm{s}}^2$, ${\sigma }_{\omega }=1 \mathrm{deg}/\mathrm{s}$ and $w_{k-1}~\mathcal{N}\left(·;0,I_3\right)$.

The initial motion states and survival durations of the four targets in the simulation scenario are presented in

Table 1. The initial states of targets.

Target Number	Initial State	Survival Time
1	${\left[\begin{array}{ccccc}-1507.4 \mathrm{m}& 11 \mathrm{m}/\mathrm{s}& 256.8 \mathrm{m}& 10 \mathrm{m}/\mathrm{s}& -1°/\mathrm{s}\end{array}\right]}^T$	$\left[0, 100\right] \mathrm{s}$
2	${\left[\begin{array}{ccccc}-255.9 \mathrm{m}& 20 \mathrm{m}/\mathrm{s}& 1011.4 \mathrm{m}& 3 \mathrm{m}/\mathrm{s}& -0.67°/\mathrm{s}\end{array}\right]}^T$	$\left[10, 80\right] \mathrm{s}$
3	${\left[\begin{array}{ccccc}246.1 \mathrm{m}& 11 \mathrm{m}/\mathrm{s}& 738.9 \mathrm{m}& 5 \mathrm{m}/\mathrm{s}& 0.5°/\mathrm{s}\end{array}\right]}^T$	$\left[35, 100\right] \mathrm{s}$
4	${\left[\begin{array}{ccccc}-242.6 \mathrm{m}& 993.2 \mathrm{m}/\mathrm{s}& 993.2 \mathrm{m}& -12 \mathrm{m}/\mathrm{s}& 1°/\mathrm{s}\end{array}\right]}^T$	$\left[40, 100\right] \mathrm{s}$

, where the target survival probability set to $p_S=0.99$.

The scene consists of four sensors observing simultaneously, including two complete measurement sensors (radar) and two incomplete measurement sensors (infrared). The observation process for all sensors lasts for $K=100 \mathrm{s}$. The measurement equation for the complete measurement sensors is

$$\begin{array}{ll}{z}_{k,m}^s& =h\left(x_{k,i},x_{k,s}\right)+{\beta }_{k,s}^c+{\upsilon }_{k,s}^c\\ & =\left[\begin{array}{c}\arctan \left(\raisebox{1ex}{$y_{k,i}-y_{k,s}$}\!\left/ \!\raisebox{-1ex}{$x_{k,i}-x_{k,s}$}\right.\right)\\ \sqrt{{\left(x_{k,i}-x_{k,s}\right)}^2+{\left(y_{k,i}-y_{k,s}\right)}^2}\end{array}\right]+{\beta }_{k,s}^c+{\upsilon }_{k,s}^c\end{array}$$

(54)

and the measurement equation for the incomplete measurement sensors is

$$\begin{array}{ll}{z}_{k,m}^s& =h\left(x_{k,i},x_{k,s}\right)+{\beta }_{k,s}^{\mathrm{In}}+{\upsilon }_{k,s}^{\mathrm{In}}\\ & =\left[\arctan \left(\raisebox{1ex}{$y_{k,i}-y_{k,s}$}\!\left/ \!\raisebox{-1ex}{$x_{k,i}-x_{k,s}$}\right.\right)\right]+{\beta }_{k,s}^{\mathrm{In}}+{\upsilon }_{k,s}^{\mathrm{In}}\end{array}$$

(55)

where $x_{k,s}={\left[x_{k,s},{\dot{x}}_{k,s},y_{k,s},{\dot{y}}_{k,s}\right]}^T$ represents the sensor state, ${\upsilon }_{k,s}^c~\mathcal{N}\left(\cdot ;0,{\sigma }_s^2\right)$ is the noise associated with complete measurements, and ${\sigma }_s^2=\left[\begin{array}{cc}{\sigma }_{\theta }^2& 0\\ 0& {\sigma }_{\rho }^2\end{array}\right]$. ${\upsilon }_{k,s}^{\mathrm{In}}~\mathcal{N}\left(\cdot ;0,{\sigma }_{\theta }^2\right)$ is the noise associated with incomplete measurements, and ${\sigma }_{\theta }=0.5°$, ${\sigma }_{\rho }=10 \mathrm{m}$. The parameters of the sensors are listed in

Table 2. The initial states and parameters of the sensors.

Sensor Number	Sensor State	Biases	Observation Time
1	${\left[1500, 0, 0, 0\right]}^T \mathrm{m}$	${\left[1°, -30 \mathrm{m}\right]}^T$	$\left[0, 100\right] \mathrm{s}$
2	${\left[0, 0, 2000, 0\right]}^T \mathrm{m}$	${\left[-1°, 30 \mathrm{m}\right]}^T$	$\left[0, 100\right] \mathrm{s}$
3	${\left[-1500, 0, 0, 0\right]}^T \mathrm{m}$	${\left[0.1°\right]}^T$	$\left[0, 100\right] \mathrm{s}$
4	${\left[0, 0, 0, 0\right]}^T \mathrm{m}$	${\left[-0.1°\right]}^T$	$\left[0, 100\right] \mathrm{s}$

.

The covariance matrix of the process noise for the two complete measurement sensors is set to $B_{k,i}^c=\left[\begin{array}{cc}{\left(0.1 \mathrm{deg}\right)}^2& 0\\ 0& {\left(1 \mathrm{m}\right)}^2\end{array}\right]$. The covariance matrix of the process noise for the two incomplete measurement sensors is set to $B_{k,i}^{\mathrm{In}}={\left(0.1 \mathrm{deg}\right)}^2$. The weights of the Gaussian components for the newborn target intensity are $w_{\gamma }^1=w_{\gamma }^2=0.02$ and $w_{\gamma }^3=w_{\gamma }^4=0.03$. The augmented mean ${\overline{m}}_{\gamma }^l$ for newborn targets is shown in

Table 3. Mean of newborn components.

Gaussian Component	${\overline{\mathit{m}}}_{\gamma }^l={\left[x_{\gamma }^l, {\dot{x}}_{\gamma }^l, y_{\gamma }^l, {\dot{y}}_{\gamma }^l, {\omega }_{\gamma }^l, {\theta }_{\gamma ,1}^l, r_{\gamma ,1}^l, {\theta }_{\gamma ,2}^l, r_{\gamma ,2}^l, {\theta }_{\gamma ,3}^l, {\theta }_{\gamma ,4}^l\right]}^T$
1	${\overline{m}}_{\gamma }^1={\left[-1500 \mathrm{m}, 0, 250 \mathrm{m}, 0, 0, 1°, -30 \mathrm{m}, -1°, 30 \mathrm{m}, 0.1°, -0.1°\right]}^T$
2	${\overline{m}}_{\gamma }^2={\left[-250 \mathrm{m}, 0, 1000 \mathrm{m}, 0, 0, 1°, -30 \mathrm{m}, -1°, 30 \mathrm{m}, 0.1°, -0.1°\right]}^T$
3	${\overline{m}}_{\gamma }^3={\left[250 \mathrm{m}, 0, 750 \mathrm{m}, 0, 0, 1°, -30 \mathrm{m}, -1°, 30 \mathrm{m}, 0.1°, -0.1°\right]}^T$
4	${\overline{m}}_{\gamma }^4={\left[1000 \mathrm{m}, 0, 1500 \mathrm{m}, 0, 0, 1°, -30 \mathrm{m}, -1°, 30 \mathrm{m}, 0.1°, -0.1°\right]}^T$

, with the corresponding variance set to ${\overline{P}}_{\gamma }^l=\mathrm{diag}{\left(50 \mathrm{m}, 50 \mathrm{m}, 50 \mathrm{m}, 50 \mathrm{m}, 6°/\mathrm{s}, 0.1°,10 \mathrm{m}, 0.1°, 10 \mathrm{m}, 0.01°, 0.01°\right)}^2,$ $l=1,2,3,4$.

The clutter follows a Poisson distribution and is uniformly distributed in the region of size $\left[-\pi ,\pi \right]\mathrm{rad}\times \left[0,2000\right] \mathrm{m}$. To prevent unlimited growth of Gaussian mixture components, filter parameters are set with a maximum limit of $L_{max}=100$ for Gaussian mixture components, a pruning threshold of $T_{pruning}={10}^{-5}$, and a merging distance of $T_{merge}=4$. The filtering performance is assessed using the OSPA distance, which provides a comprehensive measure considering both the errors in target state estimation and target cardinality estimation. In our study, we specifically set the order parameter as p = 1 and the penalty factor as c = 100 to calculate the OSPA distance. All parameters for the heterogeneous augmented GM-PHD filter are the same as those for the standard heterogeneous multi-sensor GM-PHD filter and have similar birth target intensities. To validate the HSAR-GM-PHD-R and the HSAR-GM-PHD-AO algorithms under different detection probabilities and clutter intensities, the simulation scenarios are set as shown in

Table 4. Simulation scenarios.

Scenario	Detection Probability	Clutter Intensity
1	$p_{D,k}^1=p_{D,k}^2= p_{D,k}^3= p_{D,k}^4=0.9$	${\lambda }_1={\lambda }_2={\lambda }_3={\lambda }_4=10$
2	$p_{D,k}^1=0.8, p_{D,k}^2=0.7, p_{D,k}^3=0.99, p_{D,k}^4=0.9$	${\lambda }_1={\lambda }_2={\lambda }_3={\lambda }_4=10$
3	$p_{D,k}^1=p_{D,k}^2=p_{D,k}^3=p_{D,k}^4=0.9$	${\lambda }_1=20, {\lambda }_2=40 , {\lambda }_3=5, {\lambda }_4=10$
4	$p_{D,k}^1=0.8, p_{D,k}^2=0.7, p_{D,k}^3=0.99, p_{D,k}^4=0.9$	${\lambda }_1=20, {\lambda }_2=40 , {\lambda }_3=5, {\lambda }_4=10$

.

5.2. Simulation Results

Scenario 1: Same detection probability and clutter intensity.

The detection probability for each sensor is $p_{D,k}=0.9$, and the clutter intensity is $\lambda =10$. The measurement results for sensor 1 are shown in Figure 6.

The HSAR-GM-PHD-R and HSAR-GM-PHD-AO algorithms proposed in this paper address sensor bias and target state estimation within the framework of random finite sets. In the simulated scenarios defined in this study, the angle information obtained from the infrared sensor is used to perform sequential EKF updates on the fusion state of corresponding targets observed by the radar sensor, resulting in the final fused target state. It is crucial to emphasize that the target state obtained by the algorithms proposed in this paper is the result after bias elimination. Moreover, to mitigate the impact of sensor performance on sequential fusion results, the HSAR-GM-PHD-AO algorithm conducts real-time performance evaluation of the infrared and radar sensors separately, enabling the determination of the optimal fusion order and achieving stable target tracking.

In the case of the same detection probability and clutter intensity, the performance comparison between the HSAR-GM-PHD-R algorithm and the HSAR-GM-PHD-AO algorithm is shown in Figure 7 and Figure 8.

Figure 7 and Figure 8 demonstrate that the target trajectories obtained from the HSAR-GM-PHD-R and HSAR-GM-PHD-AO algorithms closely align with the true target trajectory. When the detection probability and clutter intensity are the same, both algorithms exhibit comparable registration performance. These algorithms effectively mitigate sensor measurement biases and enhance target tracking fusion performance.

To illustrate the real-time estimation performance of bias in both algorithms, Figure 9 presents the root mean square error (RMSE) curves for sensor bias estimation in 100 Monte Carlo experiments.

By examining the RMSE curves of bias estimation for the two proposed algorithms, it is evident that the bias estimation algorithm typically converges within 10 s. The RMSE for angular estimation is approximately 1/20 of the set angular bias, while the RMSE for distance estimation is approximately 1/60 of the set distance bias. This level of accuracy in bias estimation ensures high-precision real-time operation of the fusion system. To compare and validate the performance of the two proposed algorithms under equal detection probability and clutter intensity for each sensor, Figure 10 and Figure 11 display the real-time estimation results for the number of targets (cardinality) and OSPA distances for both algorithms, respectively. Furthermore, this study focuses on analyzing the performance during the time interval with the maximum number of targets, specifically from 40 s to 80 s.

As depicted in Figure 10 and Figure 11, along with

Table 5. Performance comparison (same detection probability and clutter intensity).

Algorithm	Average OSPA Distance from 40 s and 80 s	Average Target Cardinality Estimation from 40 s and 80 s
HSAR-GM-PHD-AO	8.84	4.08
HSAR-GM-PHD-R	8.89	4.11

, when the detection probability and clutter intensity are equal for all sensors, both algorithms exhibit comparable performance in estimating the number of targets and OSPA distance. The target number estimates for both algorithms remain stable around the true values with minor fluctuations, and the OSPA distances consistently fall within a relatively low range. Specifically, for the HSAR-GM-PHD-AO algorithm, the average target number estimate between 40 s and 80 s is 4.08, deviating from the true target number by only 0.08. Moreover, the average OSPA distance during this period is 8.84. Similarly, for the HSAR-GM-PHD-R algorithm, the average target number estimate between 40 s and 80 s is 4.11, deviating from the true target number by 0.11, with an average OSPA distance of 8.89. Since the final fusion filtering results are unaffected by the sensor fusion order under the conditions of equal detection probability and clutter intensity for all sensors, both algorithms demonstrate similar registration and fusion performance.

Scenario 2: Different detection probabilities, same clutter intensity.

Figure 12 and Figure 13 present the comparison of target number estimation and OSPA distance between the RHSA-GM-PHD-R algorithm and the AOHSA-GM-PHD-R algorithm under different detection probabilities but same clutter intensity.

Figure 12 and Figure 13, along with

Table 6. Performance comparison (different detection probabilities, same clutter intensity).

Algorithm	Average OSPA Distance from 40 s and 80 s	Average Target Cardinality Estimation from 40 s and 80 s
HSAR-GM-PHD-AO	11.23	4.12
HSAR-GM-PHD-R	16.57	4.21

, demonstrate notable differences in target number estimation and OSPA distance estimation between the HSAR-GM-PHD-R algorithm and the HSAR-GM-PHD-AO algorithm when considering different detection probabilities but the same clutter intensity. The HSAR-GM-PHD-AO algorithm exhibits target number estimates that closely align with the true values, with minimal fluctuations, and maintains relatively low OSPA distances. Conversely, the HSAR-GM-PHD-R algorithm tends to overestimate the number of targets, resulting in more pronounced fluctuations around the true values and higher OSPA values. Specifically, in the 40 s–80 s time period, the HSAR-GM-PHD-AO algorithm achieves an average target number estimate of 4.12, deviating by 0.12 from the true target number. The average OSPA distance during this period is 11.23. In contrast, the HSAR-GM-PHD-R algorithm yields an average target number estimate of 4.21, deviating by 0.21 from the true target number, with a notably higher average OSPA distance of 16.57. The differences in performance arise from the fact that, when varying detection probabilities across different sensors, the sequential fusion order of different sensors can have a significant impact on the final fusion result. The HSAR-GM-PHD-AO algorithm proposed in this study efficiently plans the optimal fusion order for sensors in real time, achieving superior fusion results. Specifically, the HSAR-GM-PHD-AO algorithm achieves a 32.2% reduction in OSPA distance compared to the HSAR-GM-PHD-R algorithm. Therefore, the proposed HSAR-GM-PHD-AO algorithm outperforms the HSAR-GM-PHD-R algorithm in overall performance.

Scenario 3: Same detection probability, different clutter intensities.

Figure 14 and Figure 15 illustrate the performance comparison between the HSAR-GM-PHD-R algorithm and the HSAR-GM-PHD-AO algorithm when the detection probability is the same but the clutter intensity varies.

When the detection probability is the same but the clutter intensity varies, both the HSAR-GM-PHD-R algorithm and the HSAR-GM-PHD-AO algorithm exhibit minor differences in target number estimation and OSPA distance estimation, as shown in Figure 14 and Figure 15 and summarized in

Table 7. Performance comparison (same detection probability, different clutter intensities).

Algorithm	Average OSPA Distance from 40 s and 80 s	Average Target Cardinality Estimation from 40 s and 80 s
HSAR-GM-PHD-AO	8.48	4.13
HSAR-GM-PHD-R	8.85	4.13

. The estimated target number fluctuates significantly around the true value in both algorithms. For the HSAR-GM-PHD-AO algorithm, the average estimated target number between 40 s and 80 s is 4.13, with a deviation of 0.13 from the true target number, and the average OSPA distance is 8.48. Similarly, the HSAR-GM-PHD-R algorithm yields an average estimated target number of 4.13 with a deviation of 0.13 from the true target number, and an average OSPA distance of 8.85. The varying clutter intensities reflect differences in sensor performance, which can slightly impact the final fusion results, although to a lesser extent than detection probability. The proposed HSAR-GM-PHD-AO algorithm efficiently plans the optimal fusion order of sensors in real time, achieving optimal sequential fusion. It achieves a 4.18% lower OSPA distance compared to the HSAR-GM-PHD-R algorithm in the case of varying clutter intensities. Therefore, the HSAR-GM-PHD-AO algorithm demonstrates slightly improved performance compared to the HSAR-GM-PHD-R algorithm when clutter intensity varies among sensors.

Scenario 4: Different detection probabilities and clutter intensities.

Figure 16 and Figure 17 display the performance comparison between the HSAR-GM-PHD-R and HSAR-GM-PHD-AO algorithms when both the detection probability and clutter intensity are different.

As shown in Figure 16 and Figure 17, and

Table 8. Performance comparison (different detection probabilities and clutter intensities).

Algorithm	Average OSPA Distance from 40 s and 80 s	Average Target Cardinality Estimation from 40 s and 80 s
HSAR-GM-PHD-AO	11.58	4.05
HSAR-GM-PHD-R	17.72	3.81

, when detection probability and clutter intensity are different, the two algorithms exhibit significant differences in target number estimation and OSPA distance estimation. In comparison to the HSAR-GM-PHD-AO algorithm, the HSAR-GM-PHD-R algorithm’s target number estimation fluctuates dramatically around the true value. Specifically, the HSAR-GM-PHD-AO algorithm has an average target number estimation of 4.05 during the 40 s–80 s time interval, with a deviation of 0.05 from the true target number. At the same time, the average OSPA distance is 12.58. In contrast, the HSAR-GM-PHD-R algorithm has an average target number estimation of 3.81 during the 40 s–80 s time interval, with a deviation of 0.19 from the true target number. The average OSPA distance is 17.72. Because different sensors have varying detection probability and clutter intensity, their performance differs significantly. Consequently, the sequential fusion order of sensors has a considerable influence on the final fusion results, surpassing the impact of individual detection probability or clutter intensity. The simulation results demonstrate that the proposed HSAR-GM-PHD-AO algorithm can efficiently determine the optimal fusion order for sensors in real time, achieving optimal sequential fusion. In this regard, the HSAR-GM-PHD-AO algorithm reduces the OSPA distance relative to the HSAR-GM-PHD-R algorithm by 34.65%.

In summary, the simulation results from the four scenarios indicate that both detection probability and clutter intensity have a certain impact on fusion performance. Particularly, when the detection probabilities of the individual sensors differ, the performance differences between the two proposed algorithms become more significant. This is mainly because the detection probability significantly influences the observation quality of the sensors, making the fusion results more sensitive to the sequential fusion order. In contrast, clutter intensity has a relatively minor impact on the sequential fusion order, resulting in subtle differences between the two proposed algorithms in scenarios with varying clutter intensities.

Our proposed algorithms extend the spatial registration algorithm within the RFS framework to heterogeneous sensor scenarios for the first time. To assess the performance of our proposed algorithm, we compare the HSAR-GM-PHD-AO algorithm with the heterogeneous sensor spatial registration method based on significant targets. The results in

Table 9. Reduction rate of RMSE.

Sensor Number	Rate (%)		Observation Environment
	θ	R	Detection Probability	Clutter Intensity
S1	7.2	10.5	$p_{D,k}^1=0.8$	${\lambda }_1=20$
S2	9.7	13.1	$p_{D,k}^2=0.7$	${\lambda }_2=40$
S3	7.4		$p_{D,k}^3=0.99$	${\lambda }_3=5$
S4	10.1		$p_{D,k}^4=0.9$	${\lambda }_4=10$
Average	8.6	11.8

indicate that, under varying detection probabilities and clutter intensities, the HSAR-GM-PHD-AO algorithm achieves a substantial decrease of 11.8% in distance bias RMSE and 8.6% in angle bias RMSE, significantly enhancing the accuracy of bias estimation. This improvement can be attributed to our algorithm’s ability to avoid complex association relationships and leverage comprehensive multi-target information.

6. Conclusions

This paper addresses spatial registration in heterogeneous multi-sensor multi-target scenarios under the RFS framework. We propose two registration algorithms: HSAR-GM-PHD-R and HSAR-GM-PHD-AO. Both algorithms linearize the measurement equation by introducing pseudo-measurements and achieve a decoupled estimation of target states and measurement biases through iterative recursion of the multi-sensor GM-PHD. They also utilize EKF sequential filtering techniques for precise registration of data from different dimensions of heterogeneous sensors. To mitigate the impact of fusion order on registration and fusion accuracy, the HSAR-GM-PHD-AO algorithm employs an OSPA metric to optimize the fusion order and enhance registration tracking performance. The simulation results demonstrate that both proposed registration algorithms can effectively estimate the target position states and measurement biases of heterogeneous sensors. Additionally, under varying probabilities and clutter intensities, the HSAR-GM-PHD-AO algorithm consistently exhibits outstanding performance regardless of the sensor fusion order. Compared to the heterogeneous sensor spatial registration method based on significant targets, the average RMSE of distance and angle biases decreased by 11.8% and 8.6%, respectively, effectively enhancing the accuracy of bias estimation.

In wide-area detection scenarios, it is possible for the measurement data from different sensors to arrive at the fusion center at different times. However, our current method does not address the issue of time synchronization among the sensors. Therefore, in our future work, we plan to integrate time synchronization and spatial registration into a unified framework for sensor registration.