Joint Iterative Satellite Pose Estimation and Particle Swarm Optimization

Abstract

Satellite pose estimation (PE) is crucial for space missions and orbital maneuvering. High-accuracy satellite PE could reduce risks, enhance safety, and help achieve the objectives of close proximity and docking operations for autonomous systems by reducing the need for manual control in the future. This article presents a joint iterative satellite PE and particle swarm optimization (PE-PSO) method. The PE-PSO method uses the number of batches derived from satellite PE as the number of particles and keeps the number of epochs from the satellite PE process as the number of epochs for PSO. The objective function of PSO is the training function of the implemented network. The output obtained from the previous objective function is applied to update the new positions of the particles, which serve as the inputs of the current training function. The PE-PSO method is tested on synthetic Soyuz satellite image datasets acquired from the Unreal Rendered Spacecrafts On-Orbit Datasets (URSOs) under different preset hyperparameters. The proposed method significantly reduces the incurred loss, especially during the batch-processing operation of each epoch. The results illustrate the accuracy improvement attained by the PE-PSO method over epoch processing, but its time consumption is not distinct from that of the conventional method. In addition, PE-PSO achieves better performance by reducing the mean position estimation error by 13.1% and the mean orientation estimation error on the testing dataset by 29.1% based on the pretrained weights of Common Objects in Context (COCO). Additionally, PE-PSO improves the accuracy of the Soyuz_hard-based weight by 7.8% and 0.3% in terms of the mean position estimation error and mean orientation estimation error, respectively.

1. Introduction

Satellite pose estimation (PE) plays a crucial role in space technology and is often applied in spacecraft formation flying (SFF) [1], close proximity operations [2], relative navigation [3], on-orbit maintenance (OOM) [4], on-orbit servicing (OOS) [5], and active space debris removal [6]. The precision of vision-based PE methods depends on several factors, including the solar illumination conditions and the diffuse reflection of the Earth’s background, which form “a complex spatial environment”. Due to the satellite light level constraint imposed by a small volume and the use of a monocular camera with a low power consumption level [7], space object sensing technology has lower payload satellite requirements, but it is more often applied in special on-orbit space environments. Vision-based navigation, which is built on RGB image detection technology, has advantages in terms of its high accuracy and robustness and has recently attracted the attention of several scholars [8,9,10,11]. Most of the existing research has presented different methods or techniques with the same objective of enhancing the performance of PE. The challenge concerning this research problem is related to the lighting environment in space, which is a constraint that affects camera imaging and the relationship between the spatial position information of a satellite and its orientation characteristics. Therefore, some researchers are working on using different kinds of cameras for relative noncooperative space target PE; examples include visual cameras [12], infrared cameras [13], light detection and ranging (LiDAR) sensors [14], and time-of-flight (TOF) cameras [15,16]. Owing to the limited amount of light in space, active sensors generally have greater robustness than other sensors do. Therefore, LiDAR and TOF cameras have attracted widespread attention from researchers. For example, Ref. [15] presented the projected model of a TOF camera and proposed a new calibration model. The three utilized modules, including a TOF data processing module, a registration module, and a model mapping module, were applied to overcome the above problems with the smallest position and orientation errors.

However, vision-based navigation methods formed via the implementation of RGB images have increased in popularity since the development of artificial intelligence. Noncooperative object PE based on optical images can be described as a six-degree-of-freedom (6-DOF) PE problem in computer vision tasks. The research conducted in [17,18] focused on applying key points for PE. Their key-point detection deployment method involves information translation procedures for the object of interest by utilizing perspective-n-points (PnPs) and random sample consensus (RANSAC). Additionally, the PnP and RANSAC frameworks have been implemented [19] to enhance the satellite PE performance achieved by the procedure of a space target landmark regression network, which uses the information derived from the proposed deep 1D landmark representation that encodes the horizontal and vertical pixel coordinates of a landmark as two independent 1D vectors. Ref. [20] demonstrated a binocular PE model based on a self-supervised transformer network (STN) by generating training samples under various conditions and then implementing a combination of the SIFT and a convolutional neural network (CNN) for each generated feature. The feedforward network employed in the transformer of their proposed method was replaced with a global average pooling layer. Another study implemented GoogLeNet in their experiment with some modifications. By deploying a novel loss function, their method achieved better performance than that of the existing approaches [10]. FilterformerPose [21] is a PE network that has a CNN as its backbone and feeds its information for mapping a translation and orientation regression network; the authors devised a filter-based transformer encoder model and constructed a hypernetwork-like design based on the self-attention filtering mechanism. This approach has the ability to remove noise and generate adaptive weight information for the utilized deep learning model. Ref. [22] also presented a deep learning approach for satellite PE with the condition of knowing 3D models of the satellite. They proposed a CNN combined with 3D prior knowledge, which was expressed as a 3D model in the form of a point cloud, and used a special loss function that was more suitable for satellite PE tasks to attain desirable performance.

Many studies have examined the joint use of two techniques [23,24,25,26,27], which results in better results every iteration and can improve the final accuracy of the combined approach. The benefit of the iterative technique, which is the first process, is that it can obtain precise results before passing them to the second process. This means that the model parameters can be updated on the fly.

In the vision-based learning domain, many papers have discussed input image uncertainty [23,28,29,30,31]. Researchers have presented different ways to overcome such uncertainty or perform weakly supervised object detection. For the first time, Ref. [23] proposed a joint or cross-task enforcement method for use between weakly supervised object detection and segmentation tasks with a multitask learning scheme; this approach uses the failure patterns of each sub-method to complement the learning process of the other sub-method. Refs. [28,29,30] presented a multiple-instance learning (MIL) method with an additional technique to improve the accuracy of weakly supervised detection. Intersecting vectors [31] use binocular vision to reconstruct a straight line and, based on the direction vector and intersection of the straight line, solve the pose of the measured target in the measurement coordinate system, and obtain the initial PE value.

The optimization algorithm has been widely used for several approaches. Ref. [32] introduce distributed optimization methods for multiagent systems by using the partial information as an objective function. The result shows the effectiveness of the design. In the context of unmanned aerial vehicle (UAV) research, the gradient-free optimization algorithm has been implemented for the cooperative source-seeking problem using a group of quadrotors equipped with sensors for source scalar field measurement with limited communications [33].

Particle swarm optimization (PSO) was introduced in 1995 by Kennedy and Eberhart [34]. It has been implemented in several fields of study, including PE. Ref. [35] applied PSO to solve the ambiguity introduced by uncertainty at the locations of landmarks on the basis of the assumption that each pose hypothesis is represented by a particle. The quaternion-based PSO (Q-PSO) approach was presented in [36]. It fits a 3D model of the target object to depth images. The fitness function of this method is based on depth information only, and the quaternion formulation avoids singularities and the need for conversions between different rotation representations. PSO also implements adjustments for the regularization parameters and kernel parameters of a support vector machine [37] to improve the accuracy of facial detection. Ref. [38] applied PSO to impose kinematics constraints on the results of CNNs for implementing hierarchical hybrid hand PE. PSO was also used to introduce a cooperative target for PE with monocular vision [39], according to which the mathematical relationship between the distributed configurations of cooperative targets and the measurement accuracy was derived on the basis of the PnP principle.

In the experiment conducted herein, a novel joint iterative PE and PSO method is introduced with the aim of improving the accuracy of satellite PE, which has great benefits for the development of space technology such as close proximity operations, active space debris removal, and on-orbit servicing. The main contributions of this work are as follows.

(1)

A new end-to-end satellite PE method is proposed; this approach jointly iterates between deep learning for the PE framework and PSO. This can improve the effectiveness of the model in terms of translating the position and orientation information of the target satellite;

(2)

The design of the position parameters of the PSO algorithm, which is effectively integrated with deep learning in the PE framework, reduces the error induced by complex input images, and improves the accuracy of the satellite pose inference process;

(3)

The proposed method is verified by testing it in different downstream computer vision tasks.

The rest of the paper is organized as follows. In Section 2, the related works that briefly explain the deep learning technique for estimating spacecraft poses from photorealistic renderings [11] and the PSO algorithm is presented. Section 3 describes the utilized materials and the proposed method, which is the joint iterative method including deep learning-based spacecraft PE from photorealistic renderings and PSO, for improving the accuracy of the constructed model. The results are illustrated in Section 4. Conclusions and suggestions for future work are presented in Section 5 and Section 6, respectively.

2. Related Works

2.1. Deep Learning-Based Spacecraft Pose Estimation from Photorealistic Renderings

The deep learning framework that is applied in the experiment of this work is derived from [11]. This deep learning framework involves end-to-end PE, the drawback of which is that it does not handle multiple objects concurrently. The end-to-end PE method involves feeding images into the model, which directly computes the pose information of interest. Another method is based on the key points of the target object. However, the end-to-end PE method should be employed whenever possible to mitigate the computational overhead and inference inefficiency of other approaches.

This deep learning framework is for pose estimation. It is based on orientation soft classification, which enables modeling orientation ambiguity as a mixture of Gaussians which is a proposed method in [11] by performing continuous orientation estimation via classification with soft assignment coding [40]. This deep learning framework used ResNet architecture with pretrained weights as its network backbone. The last fully connected layer and the global average pooling layer of the conventional network were removed and retained for spatial feature resolution, effectively leaving only one pooling layer in the second layer. The global pooling layer was replaced by one extra 3 × 3 convolution with a stride of 2 (bottleneck layer) to compress the CNN features. For 3D location estimation, the framework adopts a simple regression branch with two fully connected layers. It minimizes the relative error via its loss function, which was developed on its own to be suitable for the constructed framework, instead of the Euclidean distance measure. Another study presented a mathematical model for the loss function of spacecraft PE [10,22]. This deep learning framework has also been studied to find alternative modifications or adjustments [21] because of its low number of pooling layers and good accuracy–complexity tradeoff, which are the highlights of ResNet architectures.

2.2. Particle Swarm Optimization

PSO is a metaheuristic gradient-free optimization method based on the population dynamics of particles moving with a velocity that depends on both globally optimal and particlewise optimal solutions. PSO was introduced by Kennedy and Eberhart [34] and has been shown to be effective in solving optimization problems, even in high-dimensional search spaces. Although PSO has been implemented in various studies, alternatives are being developed to achieve higher performance. For example, researchers modified the structure of PSO, such as its size or shape, to construct improved approaches, such as the hybrid PSO method [37,41,42,43].

The PSO algorithm was constructed on the basis of particles that simulate individuals in a group with two acceptance parameters, namely, positions and velocities. Assuming that the particles have a dimensionality of $N$, the position of the $i$-th particle in $N$ is denoted as $X_i$, and its velocity is $V_i$. The objective function is important in PSO, as it has a role in fitting each particle and obtaining the corresponding fitness value. Let the process obtain the best position ($p-best$) and the current position $x_i$, which represents the experience of particle actions. $g-best$ is defined as the best position found by all the particles in the entire group or the best position in $p-best$. PSO uses both $p-best$ and $g-best$ to determine the next action in the next iteration.

The PSO process begins with a random solution and then iterates to obtain the optimal solution. Each particle is updated under the conditions of $p-best$ and $g-best$ via Equations (1) and (2) to update the two parameters, as follows:

$$v_i=c_1\times rand\left(\left|\mathrm{B}\right|\right)\times \left({p-best}_i-x_i\right)+c_2\times rand\left(\left|\mathrm{B}\right|\right)\times ({g-best}_i-x_i)$$

(1)

$$x_i=x_i+v_i$$

(2)

where $i=1,\dots ,N$, $N$ is the total number of particles, and $rand\left(\left|\mathrm{B}\right|\right)$ is defined as a random number function resulting in random floats number in the half-open interval [0.0, 1.0) and has a size equal to the batch size of the input image ($\left|\mathrm{B}\right|$) in the PE algorithm which is deployed by deep learning architecture in Section 2.1. $c_1$ is the cognitive (individual) learning factor, $c_2$ is a social (group) learning factor, and $x_i$ is the current position. The expression of Equation (1) contains the magnitude and direction of the last speed, the best point between the current point and the particle, and the best point between the current point and the group. This combination determines the next action and a better $v_i$ is defined by Equation (3) as shown below:

$$v_i=\omega \times v_i+c_1\times rand\left(\left|\mathrm{B}\right|\right)\times \left({p-best}_i-x_i\right)+c_2\times rand\left(\left|\mathrm{B}\right|\right)\times ({g-best}_i-x_i)$$

(3)

where $\omega$ is a nonnegative number that is used to define the weight of the PSO process, which affects the convergence speed and convergence precision of the PSO algorithm.

3. Materials and Methods

3.1. Materials

The dataset employed in the current experiment is derived from the Unreal Rendered Spacecrafts On-Orbit Datasets (URSOs). An example of the Soyuz satellite is presented in Figure 1. This dataset consists of synthetic data from the Unreal Engine that allows the addition of realistic lighting, shadows, and other features for image generation purposes. Figure 1b explains the axis notation for a moving spacecraft frame; the green color represents the Y-axis, the red color is the X-axis, and the Z-axis is the blue one. The URSO is not only realistic but also involves a variety of data, which includes a range of spacecraft, such as space probes and satellites. The implemented dataset has a total of 5000 RGB images with resolutions of 1280 × 960 pixels and ground truths in the $[x y z q_1 q_2 q_3 q_0]$ format for all images. While $x$, $y$, $z$ are the magnitude of the relative distance in the X, Y, and Z-axes, respectively. The orientation is presented in quaternions by $q_0$ represents the real part of the quaternions and $q_1$ $q_2$ $q_3$ are the components in the vector part. The PE process also uses the quaternion as an input. However, the final results need to be converted to Euler angles for ease of understanding, and the rotation sequence is an important part of such a calculation.

The dataset possesses various positions and orientation ranges with several background images. It is important for a deep learning method to learn features under various conditions. The Soyuz_easy data package is separated into 4000 training images, 500 testing images, and 500 evaluation images.

3.2. Joint Iterative Pose Estimation and Particle Swarm Optimization (PE-PSO) Method

Dynamic sample weighting strategy is one important concept that affects the deep learning model performance [29]. The PE-PSO method has taken the concept of dynamic sample weight and joint iteration as used in [26]. The PSO algorithm deploys as the method to estimate sample weight on the fly in every epoch of spacecraft pose estimation by using deep learning. Deep learning-based spacecraft pose estimation from photorealistic renderings in Section 2.1 is used as the pose estimation method (PE) and the objective function of the PSO algorithm, while it takes the benefit from the PSO algorithm by obtaining the updated simple weight in every epoch. This manner encourages an improvement of the final pose estimation model.

The proposed method, PE-PSO, iterates between the PE and PSO processes. For the optimization problem, the slack variable ($\xi$) is added to an inequality constraint to transform it into an equality constraint. Assume that the batch size of the input training batch is $i$, the modification of the slack variable ($\xi$), which is applied to the position of the particle (${Position}_k$) in the PSO algorithm, is the following vector.

$${Position}_k=[{\xi }_0 {\xi }_{1 }\dots {\xi }_i]$$

(4)

where subscription $k$ is defined as the number of particles, whose maximum value is the number of batches in the deep learning-based PE scheme. It implies that particle $k$ has ${Position}_k$ and the ${Position}_k$ has members to be a modification of the slack variable ($\xi$) equal to the number of the input training batch ($i$).

The workflow and pseudocode of the PE-PSO method are presented in Figure 2 and Algorithm 1, respectively. In Figure 2, ${Position}_k$ is updated via the PSO process, which is depicted by the red dotted line. The objective function for PSO is defined as a PE training function as explained in Section 2.1, which results in position error (${Error}_{position,k}$), orientation error (${Error}_{orentation,k}$), and total errors (${Error}_{total,k}$). These errors are expressed as equations in the accuracy assessment section. The loss ${Loss}_k$implemented in PSO is the average of these three errors, and it is defined as the following equation.

$${Loss}_k={\displaystyle \frac{{Error}_{total,k}+{Error}_{position,k}+{Error}_{orentation,k}}{3}}$$

(5)

In the experiment, the PE process is trained with an input consisting of a batch of four images ($\left|\mathrm{B}\right|=4$) concatenated with the positions of the particles (${Position}_k$). Thus, the input for each processing iteration during batch training is expressed as follows:

$$input={[\mathrm{b}\mathrm{a}\mathrm{t}\mathrm{c}\mathrm{h}\_\mathrm{i}\mathrm{m}\mathrm{a}\mathrm{g}\mathrm{e}\mathrm{s} {Position}_k]}^T$$

(6)

In every iteration of PE, PSO recalculates the values contained in ${Position}_k$, which are part of the input. This concept is similar to the sample weight used in a general deep learning algorithm. However, it is not a predefined sample weight value that is determined before PE processing. ${Position}_k$ is not related to the sample weight and is calculated on the fly based on the PSO concepts and processes in every epoch. Therefore, these values are changed to correspond to the minimum loss of the objective function at every time step.

The velocity of a particle is computed via the following equation:

$$\begin{gathered} {Velocity}_{k,t}=\left( \omega \ast {Velocity}_{k,t-1}\right)+\left(c_1\ast r_1\ast \left({pbest}_{k,t}-{Position}_{k,t-1}\right)\right) \\ +\left(c_2\ast r_2\ast \left({gbest}_{k,t}-{Position}_{k,t-1}\right)\right) \end{gathered}$$

(7)

where $c_1$ is the local learning coefficient, $c_2$ is the global learning coefficient, and $k, t$ indicate the particle of interest and epoch, respectively. Weight (ω) is an important parameter for identifying motion during PSO. Weight (ω) controls how much of the previous velocity of the particle is retained in the current velocity. A larger weight (ω) encourages global exploration, while a smaller one promotes local exploitation. Typical values of Weight (ω) range between 0.4 and 0.9. However, it can also be dynamically adjusted during the optimization process to balance between global and local exploration [44]. In this experiment, the varied weight (ω) has been tested as present in Section 4 to seek the suitable value. $r_1$and $r_2$ are random matrices whose sizes are the batch size ($i).$ ${pbest}_{k,t}$ is the local best value, whereas ${gbest}_{k,t}$ is the global best value.

The velocity conditions of the population or a particle are as follows:

$$\begin{gathered} {Velocity}_{max}=0.1\ast \left(UB-LB\right) \\ {Velocity}_{min}=-{Velocity}_{max} \end{gathered}$$

(8)

where $UB$ and $LB$ are the upper bound and lower bound of the velocity, which are set as 20 and −20, respectively, in the current experiment.

Algorithm 1 and the workflow of the proposed method (shown in Figure 2) clearly depict the above processes. The proposed method begins with initial parameters for PE, such as the learning rate, number of epochs, number of batches, and batch size. PSO also has initial hyperparameters, namely, the positions and weights of the particles, the inertia weight damping ratio, the local learning coefficient, the global learning coefficient, and the lower and upper bounds of the population velocity. The key parameters of the joint (PE-PSO) method are the numbers of PE and PSO epochs. Moreover, it is important to set the population size or swarm size equal to the number of batches. In each epoch, PSO computes the velocity via Equation (7) with the predefined constraint imposed by Equation (8) and updates the corresponding position. The updated position is then applied as a part of the input (as Equation (6)) of the objective function to train the PE model of the spacecraft. The loss value is calculated via Equation (5) as the average of all losses involved in the training process, i.e., the total loss, location loss, and orientation loss. Consequently, the algorithm checks and updates the process based on the local (personal) and global losses by using fewer conditional operations than the previous stacking value. The weight ($\omega$) is adjusted by the inertia weight damping ratio (${\omega }_{damp}$) in every epoch. The PE-PSO process continues until the maximum number of epochs is reached.

Algorithm 1: The PE-PSO method

Input: number of batches (epochs), batch size $\left(\mathit{i}\right)$, weight $\left(\omega \right)$, inertia weight damping ratio $\left({\omega }_{damp}\right)$, local learning coefficient $\left(c_1\right)$, global learning coefficient $\left(c_2\right)$, lower bound $\left(LB\right)$ and upper bound ($UB$) of the population velocity, learning rate.

Output: The PE-PSO model.

for $\mathit{t}$  < epoch:

for $\mathit{k}$  < number of batches:

• Prepare the input and build batch logs for PE;      • Calculate the particle velocity (${Velocity}_k$) via Equation (7);      • Check for and apply velocity limits via Equation (8);      • Update the position of the particle (${Position}_k$);      • Compute ${Loss}_k$ via Equation (5);      • Check and update the positions of the local and global best particles.   Update weight as $\omega =\omega \ast {\omega }_{damp}$

3.3. Accuracy Assessment

Both the conventional model and the proposed model produce numerical outputs as their spacecraft poses $\left(Poses\right)$, which is given by seven-column datasets and is expressed as follows:

$$Poses=[x,y,z,q_1,q_2,q_3,q_0]$$

(9)

This vector is separated into two parts, namely, a position part ($[x,y,z]$) and an attitude or orientation part ($[q_0,q_1,q_2,q_3]$) that is expressed as a quaternion. The magnitudes of the X-, Y-, and Z-axes represent $x,y,z$, respectively. Regarding the orientation, $q_0$ is the real part of the quaternion, and $q_1,q_2,q_3$ are the components contained in the vector part.

The error computation is also separated into position error and orientation error components, which are calculated by the Euclidean distance measure, as described in the following questions.

$${Error}_{position}=\left|x_i-x_{gt}\right|$$

(10)

$${Error}_{orientation}=|q_i-q_{gt}|$$

(11)

where $x_i$ represents the spacecraft pose $\left(Poses\right)$ in the position part and $x_{gt}$ is the ground truth. This value corresponds with the formula used to calculate the orientation error, where $q_i$ is the output predicted from the PE model and $q_{gt}$ is the ground truth for the orientation part. The combination of ${Error}_{position}$ and ${Error}_{orentation}$ is the total error (${Error}_{total}$), which is represented in the following equation.

$${Error}_{total}={Error}_{position}+{Error}_{orientation}$$

(12)

These three kinds of errors are used for training the PE model using both the conventional and the proposed methods. Moreover, the position error is calculated in meters, whereas the orientation error transforms the quaternions in the aerospace sequence to Euler angles for model performance representation purposes.

4. Results

4.1. Training Results

First, the experiment runs for one epoch and 100 batches to evaluate the trend of the PE-PSO method. Figure 3a, Figure 3b and Figure 3c express the ${Error}_{total}$, ${Error}_{position},$ and ${Error}_{orientation}$ values produced the first 100 batches of training in the first iteration, respectively. The conventional method and the PE-PSO method are compared with $\omega =1$ and ${\omega }_{damp}=0.9$, and the Common Objects in Context (COCO) weight, where the ResNet50 architecture is used as the backbone. At the beginning, the PE-PSO method yields high values for every aspect of the error. However, these errors gradually decrease when the number of batches increases. Therefore, the hypothesis is that the PSO algorithm can work jointly with the PE process.

In the next experiments, five test cases are implemented, which are summarized as shown in

Table 1. Summary of test cases and training conditions.

Test Case No.	Training Condition
1	The conventional method
2	The PE-PSO method with $\omega =0.5$ and ${\omega }_{damp}=0.5$
3	The PE-PSO method with $\omega =0.3$ and ${\omega }_{damp}=0.5$
4	The PE-PSO method with $\omega =0.2$ and ${\omega }_{damp}=0.5$
5	The PE-PSO method with $\omega =0.1$ and ${\omega }_{damp}=0.5$

.

The weight ($\omega$) and weight damping ratio (${\omega }_{damp}$) are deployed to conduct PSO processes, with different values used for each test case. Test case number 1 is the conventional method that implements processes without the joint iterative method including the PSO algorithm. Test cases 2 to 5 are the joint processing methods with the PSO algorithm that apply a ${\omega }_{damp}$ of 0.5 and vary the value of the weight ($\omega$) to 0.5, 0.3, 0.2, and 0.1, respectively. All the experiments are implemented with 100 epochs, each epoch includes 1000 training batches, the ResNet50 structure is used as the backbone, and the model runs on a desktop computer equipped with an NVIDIA GeForce RTX 4070 GPU possessing 16 GB of VRAM.

The experiment is performed with two types of pretrained weights: (1) the COCO weight and (2) the Soyuz_hard weight [11]. ${Error}_{position}$, ${Error}_{orientation}$, and ${Error}_{total}$, which are calculated via Equations (10), (11), and (12), respectively, are presented in Figure 4. The results obtained with different training conditions exhibit different performances. For the COCO pretrained weight, the ${Error}_{total}$ of the conventional method is worse than those of the other methods, whereas PE-PSO with $\omega =0.3$ and ${\omega }_{damp}=0.5$ performs best, as shown in Figure 4a. Figure 4c shows the ${Error}_{position}$ results. Although the graph does not clearly verify that the PE-PSO method has better performance than the original method does,

Table 2. The average errors of 100 iterations of the training process with the COCO weight.

Test Case No	${\mathit{E}\mathit{r}\mathit{r}\mathit{o}\mathit{r}}_{\mathit{t}\mathit{o}\mathit{t}\mathit{a}\mathit{l}}$	Improvement	${\mathit{E}\mathit{r}\mathit{r}\mathit{o}\mathit{r}}_{\mathit{p}\mathit{o}\mathit{s}\mathit{i}\mathit{t}\mathit{i}\mathit{o}\mathit{n}}$	Improvement	${\mathit{E}\mathit{r}\mathit{r}\mathit{o}\mathit{r}}_{\mathit{o}\mathit{r}\mathit{e}\mathit{n}\mathit{t}\mathit{a}\mathit{t}\mathit{i}\mathit{o}\mathit{n}}$	Improvement	Computation Time (s)
1	6.6347	-	0.0976	-	6.5372	-	59,131.9780
2	6.6226	0.2%	0.0994	−1.9%	6.5232	0.2%	60,345.0513
3	6.3971	3.6%	0.0951	2.5%	6.3020	3.6%	59,141.1049
4	6.4770	2.4%	0.0966	1.0%	6.3803	2.4%	60,297.9563
5	6.6306	0.1%	0.0957	1.9%	6.5349	0.0%	59,284.5769

The bold in table show the best value.

presents a slight improvement. Corresponding to ${Error}_{total}$ in Figure 4e, the PE-PSO method with $\omega =0.3$ and ${\omega }_{damp}=0.5$ provides the best ${Error}_{orientation}$ results, whereas the conventional method has the highest orientation error.

Table 2 and

Table 3. The average errors of 100 iterations of the training process with the Soyuz_hard weight.

Test Case No	${\mathit{E}\mathit{r}\mathit{r}\mathit{o}\mathit{r}}_{\mathit{t}\mathit{o}\mathit{t}\mathit{a}\mathit{l}}$	Improvement	${\mathit{E}\mathit{r}\mathit{r}\mathit{o}\mathit{r}}_{\mathit{p}\mathit{o}\mathit{s}\mathit{i}\mathit{t}\mathit{i}\mathit{o}\mathit{n}}$	Improvement	${\mathit{E}\mathit{r}\mathit{r}\mathit{o}\mathit{r}}_{\mathit{o}\mathit{r}\mathit{e}\mathit{n}\mathit{t}\mathit{a}\mathit{t}\mathit{i}\mathit{o}\mathit{n}}$	Improvement	Computation Time (s)
1	5.5181	-	0.0494	-	5.4687	-	23,546.6051
2	5.5152	0.1%	0.0493	0.2%	5.4659	0.1%	23,292.7730
3	5.5140	0.1%	0.0477	3.3%	5.4662	0.0%	23,654.2353
4	5.5119	0.1%	0.0493	0.2%	5.4626	0.1%	23,548.0132
5	5.5144	0.1%	0.0487	1.3%	5.4657	0.1%	23,614.5915

The bold in table show the best value.

present the average accuracies achieved over 100 epochs of the training process with the COCO pretrained weight and the Soyuz_hard pretrained weight, respectively. Test cases 2 to 5, which are joint processes with the PSO algorithm, tend to yield better results than the conventional method does for pretrained weights. For the COCO pretrained weight, test case number 3, whose weight ($\omega$) = 0.3, performs the best for every error aspect, with a decrease of 3.6% in ${Error}_{total}$. Under the Soyuz_hard pretrained weight, test case number 4 attains the optimal performance for ${Error}_{total}$ and ${Error}_{orientation}$. The proposed method outperforms the conventional method by 0.1% in terms of both types of error, whereas test case number 3 improves ${Error}_{position}$ by 3.3% over that of the conventional method. Although Figure 4b, Figure 4d and Figure 4f, which plot the ${Error}_{total}$, ${Error}_{position}$, and ${Error}_{orientation}$ induced with the Soyuz_hard pretrained weight, respectively, clearly present outstanding performance, the improvement is reflected by the numerical values in Table 3 and is clearly described above. In addition, the computational times of the conventional method and the PE-PSO method are not different, as detailed in Table 2 and Table 3.

Table 2 shows that there is no pattern for setting the optimal number of weights $\left(\omega \right)$ and inertia weight damping ratio $\left({\omega }_{damp}\right)$. Large or small parameter values do not always result in the best performance. The results are dependent on the proper values, which are affected by several factors, such as the dataset and hyperparameters.

Moreover, the mean and standard deviation ($\sigma$) of each modified element of the slack variable ($\xi$) mentioned in Equation (4) are addressed, as presented in

Table 4. The mean and standard deviation ($\sigma$) of each modified element of the slack variable ($\xi$) in epochs 1, 50, and 100.

	Epoch 1		Epoch 50		Epoch 100
	Mean	$\mathit{\sigma }$	Mean	$\mathit{\sigma }$	Mean	$\mathit{\sigma }$
${\xi }_1$	0.9910	7.2056	0.6607	1.2528	−1.0213	1.4733
${\xi }_2$	0.9119	6.7317	1.5065	1.4761	1.0754	1.2136
${\xi }_3$	−1.3849	6.2698	−4.3527	1.4546	−4.7126	1.1934
${\xi }_4$	1.6168	7.3208	4.0583	1.2990	4.0312	1.2177

and Figure 5, by implementing test case number 3. In Figure 5, batch numbers 400, 600, and 800 are chosen for plotting, and the behaviors of these modified slack variables are presented. The Y-axis represents the modifications of the slack variables ${\xi }_1,{\xi }_2,{\xi }_3,{\xi }_4$ in Figure 5a, Figure 5b, Figure 5c, and Figure 5d, respectively, while the X-axis represents the number of epochs. This figure clearly shows that when time passes, the values of these variables tend to be stable and vary less than they did at the beginning of the interaction process. The graph in Figure 5 corresponds to the standard deviation values ($\sigma$) in Table 4. The standard deviation ($\sigma$) decreases with time, and the smallest value is obtained when the number of epochs is 100, which is the last epoch of the experiment. This finding supports the fact that higher accuracy is attained in the final case. However, slack variable element number 1 (${\xi }_1)$ is modified, where the standard deviation ($\sigma$) does not comply with the assumption because the number of iterations is not sufficient. The tradeoff between iteration time and accuracy must be carefully defined to attain appropriate performance.

4.2. Testing Result

In the testing results, the estimated mean of the position error is converted to meters, whereas the orientation is converted to Euler angles for ease of understanding and comparison [11].

Table 5. The mean estimation of position and orientation errors for the testing dataset using the COCO pretrained weights.

Test Case no.	Mean Position Estimation Error	Mean Orientation Estimation Error
1	1.3053	29.7798
2	1.2267	25.5245
3	1.1346	21.1206
4	1.2475	21.4514
5	1.2375	27.2040

The bold in table show the best value.

and

Table 6. The mean estimation of position and orientation errors for the testing dataset using the Soyuz_hard pretrained weights.

Test Case no.	Mean Position Estimation Error	Mean Orientation Estimation Error
1	0.6950	4.7611
2	0.6036	4.8252
3	0.6410	4.7488
4	0.8222	5.1743
5	0.8534	4.7671

The bold in table show the best value.

present the testing results by evaluating 500 images and implementing the tested method with the COCO pretrained weight and the Soyuz_hard pretrained weight, respectively. The testing results are presented as mean position estimation errors and mean orientation estimation errors. The numerical values in the tables indicate that test case number 3, which is the PE-PSO method with a weight $\left(\omega \right)$ of 0.3 and a weight damping ratio $\left({\omega }_{damp}\right)$ of 0.5, can achieve the best performance. It can reduce the position error from 1.3053 to 1.1346, representing an improvement of 13.1%, whereas it can improve the mean orientation estimation error under the COCO pretrained weight by 29.1%. Moreover, the PE-PSO method can enhance the performance achieved with the Soyuz_hard pretrained weight by reducing the mean position estimation error and the mean orientation estimation error by 7.8% and 0.3%, respectively. The values listed in Table 5 and Table 6 confirm that the PE-PSO method can improve accuracy or reduce the error of satellite PE whenever it is deployed with the proper weight $\left(\omega \right)$ and weight damping $\left({\omega }_{damp}\right)$ values.

4.3. Robustness Analysis

K-fold cross-validation has been implemented using the Soyuz_easy dataset, with 1000 randomly selected samples, to assess the robustness of PE-PSO in comparison to the conventional method. The selected dataset is split into 10 groups, with each group containing 100 samples. Ten groups of datasets are deployed with cross-validation techniques. If K is defined as the index of the testing dataset, with the remaining groups serving as the training dataset, the minimum value of K is 1, and the maximum value is 10. The COCO pretrained weight has been implemented to conduct a robustness analysis with the weight $\left(\omega \right)$ of 0.3 and a weight damping ratio $\left({\omega }_{damp}\right)$ of 0.5, which is shown as the best performance in the previous section. The robustness analysis is performed over 30 epochs, with the results presented in

Table 7. The mean position error for cross-validation.

K	${\mathit{E}\mathit{r}\mathit{r}\mathit{o}\mathit{r}}_{\mathit{p}\mathit{o}\mathit{s}\mathit{i}\mathit{t}\mathit{i}\mathit{o}\mathit{n}}$
	Conventional	PE-PSO	Improvement (%)
1	0.2035	0.2011	1.2%
2	0.1839	0.1827	0.7%
3	0.1982	0.1971	0.6%
4	0.1869	0.1826	2.3%
5	0.2016	0.1981	1.7%
6	0.1874	0.1889	−0.8%
7	0.1864	0.1818	2.5%
8	0.1855	0.1804	2.8%
9	0.1826	0.1784	2.3%
10	0.1873	0.1842	1.7%
Mean	0.1903	0.1875	1.5%

,

Table 8. The mean orientation error for cross-validation.

K	${\mathit{E}\mathit{r}\mathit{r}\mathit{o}\mathit{r}}_{\mathit{o}\mathit{r}\mathit{e}\mathit{n}\mathit{t}\mathit{a}\mathit{t}\mathit{i}\mathit{o}\mathit{n}}$
	Conventional	PE-PSO	Improvement (%)
1	9.5549	9.5286	0.3%
2	9.4215	9.4259	0.0%
3	9.4670	9.4544	0.1%
4	9.4421	9.4345	0.1%
5	9.5356	9.5201	0.2%
6	9.4632	9.4113	0.6%
7	9.4887	9.4882	0.0%
8	9.4406	9.4363	0.1%
9	9.4922	9.4835	0.1%
10	9.4552	9.4557	0.0%
Mean	9.4761	9.4639	0.1%

and

Table 9. The mean of total error for cross-validation.

K	${\mathit{E}\mathit{r}\mathit{r}\mathit{o}\mathit{r}}_{\mathit{t}\mathit{o}\mathit{t}\mathit{a}\mathit{l}}$
	Conventional	PE-PSO	Improvement (%)
1	9.7583	9.7298	0.3%
2	9.6054	9.6086	0.0%
3	9.6652	9.6515	0.1%
4	9.6290	9.6171	0.1%
5	9.7371	9.7181	0.2%
6	9.6506	9.6002	0.5%
7	9.6751	9.6700	0.1%
8	9.6261	9.6167	0.1%
9	9.6749	9.6620	0.1%
10	9.6425	9.6399	0.0%
Mean	9.6664	9.6517	0.2%

. Although the results for orientation error and total error, presented in Table 8 and Table 9, do not show a significant improvement, the PE-PSO method demonstrates a reduction in position error. The mean position error across all 10 test cases is reduced by 1.5%, with the highest reduction observed at K = 8, where it is mitigated by 2.8%. Moreover, the overall trend of the robustness analysis favors the PE-PSO method, particularly with further iterations, as highlighted in the previous section.

5. Conclusions

The vision-based navigation field has advanced because of the availability of research and the development of artificial intelligence, which can be applied to develop space technology. For example, vision-based navigation is applied in satellite PE because of the high precision of the close proximity, on-orbit servicing, and active space debris removal operations. Two types of vision-based navigation are available: key point-based estimation and end-to-end estimation. Key point-based satellite PE extracts key points from images via different techniques and then inputs them into a deep learning model to conduct the PE process. The drawback of this method also depends on the performance of the implemented key-point extraction technique, while other advantages include a reduction in the required processing time. For end-to-end satellite PE, the given RGB images are directly input into a deep learning model for the PE process, in which the interference of the extraction method is ignored because of the original input source. Although more computational time is required in this method than in the first approach, the overhead of computer equipment is substantial.

PE-PSO, the joint iterative deep learning-based PE and PSO method, is proposed as a novel method, where the objective is to improve the accuracy of satellite PE. The PE-PSO method is designed by adding slack variable modifications for the input data contained in every training batch of the PE algorithm. The size of the modified slack variable is adjusted so that it is equal in size to the batch size implemented in the PE algorithm, which is updated on the fly based on the concept of the PSO algorithm.

The results are presented in terms of the error reduction achieved in each epoch. In this experiment, three types of errors are analyzed: position estimation error, orientation estimation error, and total estimation error, which include position and orientation errors. PE-PSO performs well by mitigating the mean position estimation error and the mean orientation estimation error by 13.1% and 29.1% on the testing dataset, respectively, based on the COCO pretrained weight. With respect to the Soyuz_hard pretrained weight, PE-PSO achieves 7.8% and 0.3% better mean position estimation error and mean orientation estimation error reductions, respectively, than the conventional does. The PE-PSO process is based on the weight and weight damping ratio, which are variables of the PSO algorithm. Therefore, it is important to select proper values for these parameters.

6. Future Work

This research concentrates on the optimal method for achieving precise satellite PE. In future work, we will expand in two research directions. First, the satellite PE method will be tested in other environments, such as the moon, to benefit future space exploration endeavors. Second, satellite movements will be predicted based on satellite PE. For example, the alternative state estimation method can be implemented by using the PE-PSO method as a model for measuring the position and orientation of a satellite for prediction or estimation, which would be useful for satellite operations.