Motion Estimation of Non-Cooperative Space Objects Based on Monocular Sequence Images

Abstract

The motion estimation of non-cooperative space objects is an important prerequisite for successfully completing on-orbit servicing, debris removing and asteroid exploration missions. The motion estimation of non-cooperative space objects is a challenging task because of the lack of prior information about the object in unknown scenarios. In this paper, a motion estimation method of non-cooperative space objects for autonomous navigation based on a monocular sequence of images is proposed. The rotational attitude estimation of the non-cooperative space object is realized by matching the feature points of adjacent frame images, and the conditions required for translational estimation are discussed. This method has small calculation costs and runs stably, which meets the requirements of real-time computation on-board. As a result, this method implements the estimation of the complete unknown information of the object, including the attitude quaternion and angular velocity. Numerical simulations and results verify the effectiveness of the proposed method.

1. Introduction

With the continuous development of aerospace technology, faulty spacecraft on-orbit maintenance, malfunctioning satellite rescue, repairing and refueling, debris removal, asteroid exploration, etc., have become realistic problems to be solved. The motion estimation of these non-cooperative space objects is an important prerequisite for successfully completing the above missions; that is, achieving the autonomous relative navigation of non-cooperative space objects.

To improve navigation accuracy, autonomous navigation based on multi-source information fusion is applied in a specific spacecraft using microwave radars, laser radars, optical sensors, etc., as navigation sensors [1,2]. However, due to the complexity of fusion filtering algorithms and the constraints of the on-board computing ability, it is difficult to achieve real multi-source information fusion for normal satellites. As a standard configuration of a spacecraft, optical sensors have the advantages of full range applicability and rich information, which are ideal information sources for the autonomous navigation of space non-cooperative objects.

According to the application scenarios, autonomous navigation based on optical sensors can be divided into four categories. The first is autonomous relative navigation based on binocular vision. Fourie [3] proposed a method of pose estimation and 3D reconstruction of non-cooperative space objects based on binocular vision. This method has been demonstrated using real world data acquired inside the ISS [4]. Ge [5] proposed a method of motion and inertial parameter estimation of non-cooperative object on orbit using stereo vision. The measurement accuracy and range of binocular vision depend heavily on the baseline length and are limited by the size of the spacecraft itself. The method is only suitable for close range measurement, within a few meters, and cannot be applied to remote navigation in the range of a few hundred meters to tens of kilometers.

The second is autonomous relative navigation based on monocular vision and terrain matching. NASA’s Mars rovers, Curiosity [6] and Perseverance [7], have adopted image matching technology based on the terrain model of the landing area to achieve obstacle avoidance and fixed-point landing. This kind of method needs to achieve a high-precision 3D model of known objects; it needs to carry out surround measurements on the object to obtain a large number of Multiview images and send them back to earth station for processing. This kind of method has a long process period and is primarily applied to missions with known object and boundary conditions. It is not suitable for fast relative navigation missions with unknown objects.

The third is autonomous relative navigation based on monocular vision and artificial markers. In a Japanese asteroid exploration mission, Hayabusa 1 [8] and Hayabusa 2 [9] dropped luminous object markers to the surface of asteroids Itokawa and Ryugu and used a monocular optical camera to achieve relative position measurements. Due to the need to release artificial markers, this method is difficult to apply to failed satellites, space debris, etc.

The fourth is autonomous relative navigation based on SLAM. Takeishi [10] proposed a motion estimation method based on a particle filter and nonlinear optimization and applied it in Hayabusa 2. SLAM technology is mainly used in the field of ground mobile robots, and the scene is required to be static [11]. In a relative navigation application scene, the object and spacecraft are always in motion, and the traditional methods cannot be directly applied to the satellite. Fahmi [12,13,14,15,16] proposed aggregation operators on triangular cubic fuzzy numbers, which can be applied to decision-making problems. This method has application prospects in SLAM. However, the current research algorithms are primarily applied to asteroid detection scenes, and the algorithm is complex, and the computational overhead is large, which makes it difficult to meet the constraints of on-board computing ability.

To solve the above problems, a motion estimation method of non-cooperative space objects for autonomous navigation based on monocular sequence images is presented. This paper is organized as follows: Section 2 provides the proposed method. Section 3 provides the simulation results. Section 4 discusses the performance of the proposed method. In Section 5, we conclude our work.

2. Proposed Method

2.1. The Relationship between Object Motion and Camera Equivalent Motion

When the spacecraft is controlled to stare at a non-cooperative object, the object is in the field of vision of the spacecraft camera. As the motion of the spacecraft is known, the problem of estimating the object motion can be simplified as how to use fixed monocular camera sequence images to obtain the motion of the object.

This section provides the model for motion estimation of a non-cooperative object. First, we define the following reference frames (Figure 1) to make them useful for understanding the problem:

(1)

Camera-fixed frame ({O_c-X_cY_cZ_c}): A fixed camera frame centred at the position of the camera, and the frame motion with the camera.

(2)

Object-fixed frame ({O_b-X_bY_bZ_b}): A fixed object frame centred at the centroid of the object, and the frame motion with the object.

A pinhole projection model of the camera is assumed, and a feature point P_b on the object is photographed to the point P_c on the focal plane; a single feature’s measurement is expressed as:

$$\left[\begin{array}{c}{X}_c\\ Y_c\\ Z_c\end{array}\right]=R\left[\begin{array}{c}{X}_b\\ Y_b\\ Z_b\end{array}\right]+T$$

(1)

where [X_b,Y_b,Z_b]^T, [X_c,Y_c,Z_c]^T, R and T are the coordinates in the object-fixed frame ({O_b-X_bY_bZ_b}) of P_b, the coordinates in the camera-fixed frame ({O_c-X_cY_cZ_c}) of P_c, the rotation matrix and the translation vector from the object-fixed frame to the camera-fixed frame.

The pinhole projection model of the camera is expressed as:

$$Z_c·\left[\begin{array}{c}u\\ v\\ 1\end{array}\right]=\left[\begin{array}{ccc}\frac{f}{\mathrm{d}a}& 0& u_0\\ 0& \frac{f}{\mathrm{d}b}& v_0\\ 0& 0& 1\end{array}\right]·\left[\begin{array}{c}{X}_c\\ Y_c\\ Z_c\end{array}\right]$$

(2)

where u and v are the two direction image pixel coordinates of feature point P_b, u₀ and v₀ are the center pixel coordinates of the image photographed by the camera, and da, db and f are the two direction cell sizes of the focal plane and the focal length of the camera, respectively.

For a translating and rotating space non-cooperative object, we use a monocular camera fixed with an inertial reference frame to take pictures of the object. The position and attitude, relative to the camera-fixed frame {O_c-X_cY_cZ_c}, are represented by translation vector T and attitude rotation matrix R; then, the different T and R of the object motion, relative to the camera, can be obtained at different times. Similarly, the object position and attitude between two moments can be represented by another translation vector and attitude rotation matrix (Figure 2).

At the two different moments, i and i + 1, the coordinates of the feature point P on the non-cooperative object in the frame {O_c-X_cY_cZ_c} are (X_c,i, Y _c,i, Z _c,i) and (X_c,i+1, Y _c,i+1, Z _c,i+1). The attitude rotation matrices from {O_b-X_bY_bZ_b} to {O_c-X_cY_cZ_c} are R_i and R_i+1, and the translation vectors are T_i and T_i+1. The attitude rotation matrix and translation vector of the object changing from moment i to moment i + 1 are $\Delta$G_i and $\Delta$L_i. Then, the coordinate transformation relationship can be obtained:

$$\left[\begin{array}{c}{X}_{c,i}\\ Y_{c,i}\\ Z_{c,i}\end{array}\right]=R_i·\left[\begin{array}{c}{X}_b\\ Y_b\\ Z_b\end{array}\right]+T_i$$

(3)

$$\left[\begin{array}{c}{X}_{c,i+1}\\ Y_{c,i+1}\\ Z_{c,i+1}\end{array}\right]=R_{i+1}·\left[\begin{array}{c}{X}_b\\ Y_b\\ Z_b\end{array}\right]+T_{i+1}$$

(4)

$$R_i=R_{i+1}·\Delta G_i$$

(5)

$$T_{i+1}=T_i-\Delta L_i$$

(6)

where T_i, T_i+1 and $\Delta$L_i are described in the camera-fixed frame {O_c-X_cY_cZ_c}.

Then, combining Equations (3) and (4), the following expressions can be obtained:

$$\left[\begin{array}{c}{X}_{c,i+1}\\ Y_{c,i+1}\\ Z_{c,i+1}\end{array}\right]=R_{i+1}·R_i^T·\left[\begin{array}{c}{X}_{c,i}\\ Y_{c,i}\\ Z_{c,i}\end{array}\right]-R_{i+1}·R_i^T·T_i+T_{i+1}$$

(7)

Equation (7) can be written as follows:

$$\left[\begin{array}{c}{X}_{c,i+1}\\ Y_{c,i+1}\\ Z_{c,i+1}\end{array}\right]=\Delta R_i·\left[\begin{array}{c}{X}_{c,i}\\ Y_{c,i}\\ Z_{c,i}\end{array}\right]+\Delta T_i$$

(8)

where it can be denoted that:

$$\{\begin{array}{l}\Delta R_i=R_{i+1}·R_i^T\\ \Delta T_i=T_{i+1}-\Delta R_i·T_i\end{array}$$

(9)

The above observation movement process is essentially the relative movement of the camera and the object. It can be understood in two equivalent ways. The first interpretation is the real physical state; that is, the camera does not move, and the object moves. The second equivalent interpretation is to take the object as a reference. Imagine that a person stands on a moving object and observes a fixed camera. This can be equivalent to the object not moving and the camera moving, as shown in Figure 3. Therefore, $\Delta$R_i and $\Delta$T_i represent the rotation attitude matrix and translation vector of the camera-fixed frame {O_c-X_cY_cZ_c} from moment i to moment i + 1.

Furthermore, combining Equations (5), (6) and (9), the following equations can be obtained:

$$\{\begin{array}{l}\Delta R_i=\Delta G_i^T\\ \Delta T_i=\underset{A}{\underbrace{T_i-\Delta R_i·T_i}}-\Delta L_i\end{array}$$

(10)

Equation (10) provides the relationship between object motion and camera equivalent motion. Note that $\Delta$T_i consists of two parts, and the part $\underset{A}{\underbrace{T_i-\Delta R_i·T_i}}$ represents the equivalent translation vector of the camera-fixed frame caused by the object rotation.

2.2. Camera Translation and Rotation from Sequence Images

A sequence image is constructed by observing the object with a monocular camera, and the same feature point in two adjacent image frames conforms to the epipolar constraint:

$$\left[\begin{array}{ccc}{X}_{c,i+1}& Y_{c,i+1}& Z_{c,i+1}\end{array}\right]·\Delta {\tilde{T}}_i·\Delta R_i·\left[\begin{array}{c}{X}_{c,i}\\ Y_{c,i}\\ Z_{c,i}\end{array}\right]=0$$

(11)

where $\Delta {\tilde{T}}_i=\left[\begin{array}{ccc}0& -T_{z,i}& T_{y,i}\\ T_{z,i}& 0& -T_{x,i}\\ -T_{y,i}& T_{x,i}& 0\end{array}\right]$ and $\Delta T_i=\left[\begin{array}{c}{T}_{x,i}\\ T_{y,i}\\ T_{z,i}\end{array}\right]$.

Bringing Equation (2) into the above equation, the following can be obtained:

$$\left[\begin{array}{ccc}{u}_{i+1}& v_{i+1}& 1\end{array}\right]·A^{-T}·\Delta {\tilde{T}}_i·\Delta R_i·A^{-1}·\left[\begin{array}{c}{u}_i\\ v_i\\ 1\end{array}\right]=0$$

(12)

where $A=\left[\begin{array}{ccc}\frac{f}{\mathrm{d}a}& 0& u_0\\ 0& \frac{f}{\mathrm{d}b}& v_0\\ 0& 0& 1\end{array}\right]$ is the camera internal parameter matrix, and $\left[\begin{array}{c}{u}_i\\ v_i\\ 1\end{array}\right]$ is the pixel coordinates at the moment $i$.

We denote that $E=\lambda ·\Delta {\tilde{T}}_i·\Delta R_i$ and $F=A^{-T}·\lambda ·\Delta {\tilde{T}}_i·\Delta R_i·A^{-1}$, which are the essential matrix and the fundamental matrix of the stereo vision, respectively, and the eight-points method is often used to obtain their values, where $\lambda$ is a constant scale factor. After obtaining the essential matrix and the fundamental matrix by feature matching, the $\lambda ·\Delta T_i$ and $\Delta R_i$ can be obtained by performing singular value decomposition (SVD) decomposition on E.

Assume that the rotation attitude matrix is R₀ and the translation vector is T₀ at the initial moment. Then, according to Equation (7), the attitude rotation matrix $\Delta R{s}_i$ and translation vector $\Delta T{s}_i$ describing the position and attitude change of {O_c-X_cY_cZ_c} from the initial moment to moment i are obtained:

$$\{\begin{array}{l}{T}_i=\left(\underset{s=1}{\overset{i-1}{\Pi }}\Delta R_s\right)·T_0+\left(\underset{s=2}{\overset{i-1}{\Pi }}\Delta R_s\right)·\Delta T_1+\cdots +\left(\underset{s=i-1}{\overset{i-1}{\Pi }}\Delta R_s\right)·\Delta T_{i-2}+\Delta T_{i-1}\\ R_i=\Pi \Delta R_{i-1}\cdots \Delta R_2·\Delta R_1·R_0\end{array}$$

(13)

$$\{\begin{array}{l}\Delta T{s}_i=T_i-\left(\underset{s=1}{\overset{i-1}{\Pi }}\Delta R_s\right)·T_0={\displaystyle \sum _{j=1}^{i-2}\left(\underset{s=j+1}{\overset{i-1}{\Pi }}\Delta R_s\right)·\Delta T_j+\Delta T_{i-1}}\\ \Delta R{s}_i=\underset{s=1}{\overset{i-1}{\Pi }}\Delta R_s\end{array}$$

(14)

where $\Pi$ is the continuous multiplication notation, and $\Sigma$ is the continuous addition notation.

2.3. The Model for Estimating the Object Rotation State

At the initial moment of observation, the object-fixed frame $\left\{O_b{\mathit{-X}}_b{Y}_b{Z}_b\right\}$ parallel to the camera-fixed frame {O_c-X_cY_cZ_c} is established on the object; that is, the attitude rotation matrix is a unit matrix. Then, the matrix $\Delta R{s}_i$ is equal to R_i, and the transpose matrix $R_i^T$ of R_i is the attitude rotation matrix of the object, which represents the attitude rotation of the object. Therefore, it can be concluded that the rotation of the object from the observation start time to any time can be determined.

Taking the derivation of the attitude rotation matrix $R_i^T$, the object angular velocity relative to the object-fixed frame $\left\{O_b{\mathit{-X}}_b{Y}_b{Z}_b\right\}$ can be obtained:

$${\tilde{\mathsf{\omega }}}_{b,i}=-{\dot{R}}_i^T{R}_i$$

(15)

where ${\mathsf{\omega }}_{b,i}=\left[\begin{array}{c}{\omega }_{bx,i}\\ {\omega }_{by,i}\\ {\omega }_{bz,i}\end{array}\right]$ and ${\tilde{\mathsf{\omega }}}_{b,i}=\left[\begin{array}{ccc}0& -{\omega }_{bz,i}& {\omega }_{by,i}\\ {\omega }_{bz,i}& 0& -{\omega }_{bx,i}\\ -{\omega }_{by,i}& {\omega }_{bx,i}& 0\end{array}\right]$.

Furthermore, through the projection transformation of the object angular velocity in the object-fixed frame by $R_i$, the object angular velocity relative to the camera-fixed frame can be obtained:

$${\mathsf{\omega }}_{c,i}=R_i·{\mathsf{\omega }}_{b,i}$$

(16)

Similarly, the attitude quaternion of the object can be obtained through the attitude rotation matrix $R_i$:

$$q_{e,i}=quat\left(R_i\right)$$

(17)

Then, the rotation axis direction and rotation angle are:

$$\{\begin{array}{l}{\theta }_i=\pi -2·{\sin }^{-1}\left(q_{e,i}\left(4\right)\right)\\ e_i=\frac{q_{e,i}(1:3)}{\Vert q_{e,i}(1:3)\Vert }\end{array}$$

(18)

Thus far, the model for estimating the rotation state of the object has been obtained, as follows:

$$\{\begin{array}{l}{R}_i=\Pi \Delta R_{i-1}\cdots \Delta R_2·\Delta R_1\\ q_{e,i}=quat\left(R_i\right)\\ {\tilde{\mathsf{\omega }}}_{b,i}=-{\dot{R}}_i^T{R}_i\\ {\mathsf{\omega }}_{c,i}=R_i·{\mathsf{\omega }}_{b,i}\end{array}$$

(19)

2.4. Discussion on Estimating the Object Translation State

Within comparison to estimating the rotation of an object, the translation problem is more complicated. The conclusion is given by Equation (10) that the camera translation consists of two parts, and the object real translation is included in $\Delta T_i$. To clarify the composition of the translation vector $\Delta T_i$, we discuss the equivalent translation process of the camera in detail, as shown in Figure 4.

From the decomposition of the essential matrix, $\lambda ·\Delta T_i$ can be obtained. This means that the direction is known, but the distance scale is missing. The triangle constituted by $\Delta T_i$, $T_i-\Delta R_i·T_i$ and $-\Delta L_i$ cannot be determined by only one side direction. To know what information is needed to determine the triangle, step by step, some assumptions are made:

• If the size and direction of $\Delta T_i$ and $T_i-\Delta R_i·T_i$ are known, $\Delta L_i$ can be determined.
• If the size and direction of $T_i$ and $T_{i+1}$ are known, $\Delta T_i$ and $T_i-\Delta R_i·T_i$ can be determined.
• If only the direction of $T_i$ and $T_{i+1}$ is known, the ratio of $\Delta T_i$, $T_i$ and $T_{i+1}$ can be determined, that is, unified to the same scale. Then, $T_i-\Delta R_i·T_i$ can be unified to this $T_i$ scale, and the direction of $\Delta L_i$ can be determined.
• On the basis of 3, if the size of $T_i$ is also known, $\Delta L_i$ can be determined. This means that if the position and direction at the initial moment and the direction at any moment of the object centroid are known, the translation of the object can be solved.
• If only the direction of $T_i$ is known, the direction of $T_i-\Delta R_i·T_i$ can be determined. If the directions of $T_i-\Delta R_i·T_i$ and $\Delta T_i$ are not parallel, the object must have translation. It should be pointed out that this is a sufficient and unnecessary condition.

The above assumptions can guide us to obtain the relevant data from other sensors or the configuration of sensors during the application of the proposed method.

3. Results

The calculation effect of this method will be verified by mathematical simulation. Assuming that there is a free nutation satellite in orbit, it represents a typical space non-cooperative object. Before the robot operates on the orbit, it first needs to obtain the motion state of the object.

The mass characteristics and initial motion state of the satellite described in the object-fixed frame are as follows:

• The mass is $M_b=3000\left(\mathrm{kg}\right)$.
• The moment of inertia relative to the centeriod is as follows:

  $$J_b=\left[\begin{array}{ccc}5887.73& -970.31& -3193.43\\ -970.31& 12708.23& -1063.51\\ -3193.43& -1063.51& 9883.97\end{array}\right]\left(\mathrm{kg}·{\mathrm{m}}^2\right)$$
• The initial angular velocity is ${\mathsf{\omega }}_b={\left[\begin{array}{ccc}1& 0.2& 0.1\end{array}\right]}^{\mathrm{T}}\left(\circ /s\right)$.

The centroid position of the satellite described in the camera-fixed frame is as follows:$\left[\begin{array}{ccc}0& 0& 40\end{array}\right]\left(\mathrm{m}\right)$.

The internal parameter matrix of the camera is as follows:

$$A=\left[\begin{array}{ccc}2909& 0& 1024\\ 0& 2909& 1024\\ 0& 0& 1\end{array}\right]$$

The above parameter selection comes from the general satellites, and they are simply a set of horizontally comparable parameters that are randomly assumed to not affect the results. The number of feature points extracted from the object needs to be greater than eight. The camera’s internal parameter matrix is derived from an ideal camera with a focal length of 16 mm, the size of a single cell on the image plane is 5.5 microns, and the total number of pixels is 2048 by 2048.

Through the object feature recognition and matching algorithm, more than eight feature points on the object are extracted and stably tracked, as shown in Figure 5. The spatial motion trajectory of the object feature points is shown in Figure 6. The tracked object image feature points and their change trajectories are shown in Figure 7.

The attitude quaternion and angular velocity of the object are obtained by the method in this paper, as shown in Figure 8 and Figure 9, where the red dotted curve is the result estimated by the proposed method, and the blue solid curve is the real state of the object simulation.

4. Discussion

Through simulation, it can be proven in entirety that the proposed method is feasible; that is, the attitude motion of the target can be obtained by a sequence of images of the object using only the monocular camera, and the required conditions for estimating the translation of the object are analyzed. In terms of hardware resources and the motion estimation process, the consumption is low. However, the simulation only verifies the validity of the motion estimation, and the image processing algorithm is not the focus of this paper; the image processing algorithm is very important and is the premise of this method. Therefore, motion estimation can only be achieved if accurate matches of more than eight feature points can be obtained. Overall, the rotation speed and observation distance of the object, as well as the image processing algorithm, will affect the extraction accuracy of the feature points, which is another important issue and requires further research.

There are various models available, and the current study differs from others. At present, some methods of motion estimation for non-cooperative space objects need to be implemented through binocular cameras. However, on the satellite, the baseline length of the binocular camera is greatly limited. Other methods need to use microwaves or lidar, which will encounter difficulties in the equipment installation layout and consume satellite resources. Moreover, many methods use complex fusion filtering algorithms. These all reduce the application effect of onboard autonomous navigation. In contrast to other methods, which require binocular cameras or radar to measure data, this method only uses a monocular camera, which has low requirements for the configuration of the measurement sensors. Therefore, the computational complexity and processing speed are the advantages of this method. Similarly, this method has some disadvantages, such as being unable to estimate the real size of the translation and requiring high image processing accuracy. However, it is still a novel method for the motion estimation of non-cooperative space targets.

5. Conclusions

In this paper, a novel motion estimation method of non-cooperative space objects for autonomous navigation based on monocular sequence images is proposed. This method is not limited by the baseline length as in the stereo vision algorithm and has no constraints on the size and range of the object. It can be applied to asteroid exploration missions. This method does not require prior information such as the three-dimensional model of the object or artificial markers. It only uses the sequence images of the monocular camera to estimate the motion of the object. This method only uses image matching to obtain the motion estimation of the object, which avoids the problems of stability and convergence time of the filtering algorithm. Theoretically, only two images of the initial adjacent frames are needed to obtain the motion estimation of the object. This method has low calculation cost, runs stably and realizes the simultaneous modelling and estimation of the space moving object. Moreover, the algorithm only uses a basic matrix vector calculation, which is convenient for code transplantation and embedded implementation, and can well meet the requirements of real-time computation on board.