3D Pose Estimation for Object Detection in Remote Sensing Images

Abstract

3D pose estimation is always an active but challenging task for object detection in remote sensing images. In this paper, we present a new algorithm for predicting an object’s 3D pose in remote sensing images, called Anchor Points Prediction (APP). Compared to previous methods, such as RoI Transform, our object results of the final output can obtain direction information. We predict the object’s multiple feature points based on the neural network to obtain the homograph transformation relationship between object coordinates and image coordinates. The resulting 3D pose can accurately describe the three-dimensional position and attitude of the object. At the same time, we redefine the method $Io{U}_{APP}$ for calculating the direction and posture of the object. We tested our algorithm on the HRSC2016 dataset and the DOTA dataset with accuracy rates of 0.863 and 0.701, respectively. The experimental results show that the accuracy of the APP algorithm is significantly improved. At the same time, the algorithm can achieve one-stage prediction, which makes the calculation process easier and more efficient.

1. Introduction

In recent years, with the deepening of research and the improvement of computing power, deep learning has become more and more widely used in various fields. At the same time, the object detection algorithm has made great progress so far. In particular, remote sensing images has been a specific but active topic in computer vision [1,2]. Recent progresses in object detection in aerial images have benefited a lot from the R-CNN frameworks [1,3,4,5,6]. These methods use horizontal bounding boxes as the region of IoU and then rely on region-based features for category identification [2,7,8]. Faster-RCNN [4,5] leads to an elegant and effective solution where proposal computation is nearly cost-free given the detection network’s computation. A multi-stage object detection framework, the Cascade R-CNN, is proposed for the design of high-quality object detectors [6,9]. Additionally, FPN uses feature pyramids for object detection [10]; Yolt achieves object detection of high-resolution remote sensing images based on Yolo v3 [11,12]; and Yolo v3 is significantly faster than other methods in achieving the same accuracy [13]. These classic algorithms have different adaptation scenarios and greatly promote the development of this field. However, in remote sensing images, the object is often placed obliquely, so using an inclined box to detect the object will be more adaptive to the scene. These algorithms use a horizontal rectangular box to detect the object, so it does not accurately reflect the object pose of the remote sensing image to some extent. Also, these horizontal RoIs typically lead to misalignments between the bounding boxes and objects [8,14,15]. The RoI Transform algorithm locates the inclined box by predicting the rotation angle of the object box [8,16]. However, this algorithm has some problems. The first problem is that the rotation angle $\theta$ of the regression inclined box is ambiguous in most cases. This means that $\theta$ is equal to 0° and 180° corresponding to IoU is equal, but if the algorithm does not contain direction information, it will be considered the same type. The second problem is efficiency. It is a two-stage algorithm and the localization method relies entirely on Faster-RCNN [5,8]. The algorithm can only use the rectangle obtained by Faster-RCNN, and cannot use the feature information of the object region. CornerNet is a new one-stage approach to object detection by predicting the coordinates of the top-left and bottom-right points that does away with anchor boxes, which is more accurate and efficient [17,18,19,20]. However, predicting two points does not fully describe the information of the inclined box [5,21,22].

A 3D pose describes the three-dimensional pose of the camera relative to the object’s own coordinate system, not the pose of the object relative to the ground plane. Whether the reference object has z-coordinates and whether 3D information can be estimated are not the same thing. The object is on a plane and does not affect the rotation and translation of the camera observing the object relative to the three axes in three dimensions. Therefore, even objects on a two-dimensional plane will be observed in three-dimensional space out of the plane, which also has the 3D pose problem.

To solve the above problems, we propose the Anchor Points Prediction (called APP) algorithm. Different from other methods, we predict the position and attitude of the object by at least four corner points through the full convolution network, and can obtain the 3D pose by decomposing the homograph transformation matrix, and the algorithm is more efficient. The corner pooling layer used in the algorithm greatly improves the points prediction accuracy [17].

We give the correspondence between the predicted points and the available object information in

Table 1. Correspondence between predicted points and available object information [17,23].

Prediction Points	Object Information
2	RoI
4	inclined box
≥4	3D pose

, and a comparison of the traditional method and our method is shown in Figure 1. We have reason to believe that object detection by point prediction will become a new trend in the future.

2. Object Detection Based on APP

Any object detection problems can be attributed to the prediction of key points. The traditional rectangle object detection methods can be attributed to the prediction of two key points, such as Faster-RCNN, YOLO, and SSD (the upper-left and lower-right points of the rectangle) [5,21,24,25]; the 3D pose of the general object can be attributed to the prediction of eight points; the human body posture OpenPose can be attributed to the prediction of 18 key points of the human body [26]. We attribute the predicted inclined object box to a prediction of four points. The traditional inclined box detection methods have no direction information and may result in high accuracy. In addition, as shown in Figure 2, the boxes of the two objects whose center points are close but opposite in direction may cause the object to be lost in the NMS operation. The full name of NMS is non-maximum supply [27]. This method is used to search the local maximum and suppress the maximum. The purpose of this method is to eliminate redundant frames and find the best location for object detection.

Unlike traditional methods, we define a new calculation method that takes into account the overlap rate and direction consistency between the tilted boxes. Assuming there are two sets of object feature points, $\{P_{11}\dots P_{1n}\}$, $\{P_{21}\dots P_{2n}\}$, we define the $Io{U}_{APP}$ calculation formula between the two sets of points as follows:

$${IoU}_{APP}=\frac{1}{1+\alpha \sqrt{\frac{2d_{12}}{d_1+d_2}}}.$$

(1)

We use ${\mathrm{d}}_{12}={\sum }_{i=1}^n{\left(P_{1i}-P_{2i}\right)}^2$, ${\mathrm{d}}_1={\sum }_{i=1}^n{\left(P_{1i}-\overline{P_1}\right)}^2$ and ${\mathrm{d}}_2={\sum }_{i=1}^n{\left(P_{2i}-\overline{P_2}\right)}^2$. $\overline{P_1}$ and $\overline{P_2}$ are the coordinates of the center point of point set 1 and point set 2, respectively. The definition mainly considers the deviation of the coordinate offset of the corresponding point from the size of the object itself. This deviation is relative. The larger the deviation, the smaller the $Io{U}_{APP}$. It is clear that the range of values of $Io{U}_{APP}$ is the same as the original IoU definition, which is [0, 1]. The larger the $Io{U}_{APP}$, the closer the two object cells are, and the $Io{U}_{APP}$ is equal to 1 when the two object cells are completely identical; the $Io{U}_{APP}$ is infinitely close to 0 when the two object cells are very different.

The two sets of the object feature points may be the two prediction units to be combined in the object detection NMS process, or may be the similarity calculation between ground truth and the predicted values.

According to Figure 3, we can get the calculation formulas for $Io{U}_{APP}$, IoU, and $Io{U}_{RBox}$ as follows, and we can get the relationship curve as shown in Figure 4.

$$\begin{array}{cc}\hfill & {IoU}_{APP}=\frac{1}{1+8\alpha {sin}^2\frac{\theta }{2}},\alpha =1,2,3,\dots \hfill \\ \hfill & IoU=\frac{1}{{\left(1+2\frac{|cos\theta sin\theta |}{1+|cos\theta |+|sin\theta |}\right)}^2}\hfill \\ \hfill & {IoU}_{rbox}=\frac{1}{|cos\theta |+|sin\theta |}.\hfill \end{array}$$

(2)

It can be seen from the above figure that when the object position scale is constant, only the $Io{U}_{APP}$ is significantly affected by the object direction angle $\theta$, so only the $Io{U}_{APP}$ can describe the accuracy of the object direction angle. The IoU and $Io{U}_{RBox}$ are not affected by $\theta$ and cannot describe the accuracy of the object direction angle.

The mAP (mean Average Precision) obtained from the experimental data of RoI Transform is based on $Io{U}_{RBox}$ [8]. The mAP, which is used to evaluate the accuracy of object detection methods, is based on IoU between prediction boxes and ground truth boxes [28]. As can be seen from the above Figure, as long as the neural network that can recommend the horizontal box aligns with the center point, the $Io{U}_{RBox}$ is always greater than 0.5. That is, even if the direction angle prediction is wrong (predicted to be any angle from 0° to 360°), it is also hit when counting mAP, so the resulting mAP is virtually high. Thus, we proposed the solution $Io{U}_{APP}$. The $Io{U}_{APP}$ uses the method of regression coordinates to detect the object, and the method of evaluating mAP is more reasonable.

3. Anchor Points Prediction Algorithm

3.1. Neural Network Design

To predict the inclined box of the remote sensing object, we built a full convolutional network that predicts three scales in three different layers, each scale being the output by the APP of three different anchors’ array. Different anchors are used to detect objects with different aspect ratios in the image domain, as shown in Figure 5.

Our custom region layer is used to output the relative coordinates, the categories, and the information about whether the object exists or not. In the region layer, we used yolov3’s definition of anchors to implement n-weight anchor predictions based on the width and height of the object on the imaging surface. Each anchor represents a specific 2D wide height object. Following this concept, this particular 2D width and height corresponds to different APP range distributions. Each cell of the output array contains $n\times (4+1+c+2\times 4)$ output neurons. The meaning of the parameters are shown in

Table 2. The meaning of the parameters of the output.

Parameter	4	n	1	c	$2\times 4$
Description	bounding box coordinates	number of anchors	existing object	number of classes	$(\Delta x_1,\Delta y_1)$$(\Delta x_2,\Delta y_2)$ $(\Delta x_3,\Delta y_3)$$(\Delta x_4,\Delta y_4)$

.

We define the offset coordinate of point i in the range of a specific region as $\left(p_w\times \Delta x_i,p_h\times \Delta y_i\right)$. As shown in Figure 6, $p_w$ and $p_h$ are the width and height of a particular anchor. We can use Equation (3) to calculate $P_i$.

$$\{\begin{array}{l}{u}_i=C_x+\frac{1}{2}+p_w{x}_i\\ v_i=C_y+\frac{1}{2}+p_h{y}_i\end{array}$$

(3)

Then, the actual pixel offset coordinate of point i relative to the anchor boxes is $P_i=(u_i,v_i)$. In addition, we define the loss function in the training procedure as follows, and we give the meaning of each parameter in

Table 3. The meaning of the parameters of Loss function.

Parameter	${\mathit{\lambda }}_{\mathit{n}\mathit{o}\mathit{o}\mathit{b}\mathit{j}}$	${\mathit{\lambda }}_{\mathit{o}\mathit{b}\mathit{j}}$	${\mathit{\lambda }}_{\mathit{R}\mathit{O}\mathit{I}}$	${\mathit{\lambda }}_{\mathit{p}\mathit{t}\mathit{s}}$	${\mathit{P}}_{\mathit{i}}$	$\overline{{\mathit{P}}_{\mathit{i}}}$
Description	loss weight of non-object	loss weight of object	loss weight of object RoI	loss weight of each point	predicted value of the point $(u_i,v_i)$	ground truth of the point $(\overline{u_l},\overline{v_l})$

.

$$Loss(OutputLayer)=\{\begin{array}{l}{\lambda }_{nobj}{(0-Obj)}^2\mathrm{No}\mathrm{object}\mathrm{in}\mathrm{local}\mathrm{scope}\\ {\lambda }_{obj}{(1-Obj)}^2+{\lambda }_{pts}\sum _{i=1}^4{\left(P_i-{\overline{P}}_i\right)}^2+{\lambda }_{ROI}Los{s}_{ROI}\mathrm{object}\mathrm{within}\mathrm{a}\mathrm{local}\mathrm{scope}\end{array}$$

(4)

We are more focused on the positioning learning of the inclined box determined by APP, so we reduce the weight of the horizontal box. Then, we use ${\lambda }_{ROI}=0.01$ to assist learning, and we can even set it to 0 to ignore the weight of the horizontal box. $Los{s}_{ROI}$ is the regression error of the center point and width of the object RoI, following yolo v2 [12].

The principle of judging whether there is an object in a local range is to calculate the maximum IoU between the default anchors’ boxes and all the ground truth boxes in that range. If the IoU exceeds a threshold, there will be an object. The principle of discrimination here is consistent with the processing of yolo v2 and yolo v3 [12,13]. Therefore, the sum of the differences in the three-layer APP prediction result and the ground truth is

$$Loss=\sum _{\mathrm{output}\mathrm{Layer}}\mathrm{Loss}(\mathrm{Output}\mathrm{Layer}).$$

(5)

3.2. Training Procedure

Training datasets. We experimented with the DOTA dataset. The original DOTA images are high-resolution remote sensing images, which is not convenient for direct processing using the neural network. Therefore, first of all, the raw data needs to be standardized. The method we took was to randomly select an image point, then center the point, and align the center point (W/2, H/2) of the transformed image (W, H) for random affine transformation. The scale of the transformation is 0.5 to 1.5 times the original image. Then, we obtain the sample images.

Training and testing. At training, we used 80% DOTA images, and all the processed images were resized into 416*416 and sent to the neural network. After training, the remaining 20% of images were used for testing. In the training process, the choice of multiple anchors followed the strategy of yolo v3, and the process of backpropagation of the loss layer was divided into two phases.

Phase 1: Scan each output of the output layer array. According to the ground truth set and the boxes determined by predicted APP coordinates, the output region can be obtained from the maximum IoU between them. If $Io{U}_{max}$ is less than $\epsilon$, the corresponding object presence expectation output value will be set to 0 for backpropagation correction.

Phase 2: Scan the rows and columns of each GTBox, and correct the largest anchor of the IoU between the default rectangle of n anchors at this position and the GTBox. Then, set the expected value of the object field to 1, and the loss of APP is calculated according to Equation (19), and the expected value of the softmax segment is set to perform backpropagation correction.

Application. According to the four-sided output bounding box surrounded by four APP coordinates, the four points of the inclined box are further obtained by Equation (19), and the point coordinates are converted to the large image according to Equation (5).

3.3. Calculation of the Object of 3D Pose

The conventional methods often calculate the 3D pose of the object by matching the local features extracted in the 2D image with the features in the object 3D model to be detected, but these methods are not accurate enough [29]. Therefore, based on the key point coordinates of the object output from the region layer, we use the perspective transformation method to calculate the 3D pose. Figure 7 shows the computational process of the objects’ 3D pose. We use two methods, PnP [30] and homograph. The following describes these two methods.

3.3.1. PnP Method

As shown in Figure 7, we can get the coordinates$\{P_1,P_2,P_3,P_4\}$ of the four feature points of the object in the training part. These feature point coordinates are relative to the coordinates of the complete satellite image. If the image is cropped and resized, these coordinates need to be transformed to the coordinate system of the original satellite image. The inference part obtains the 3D pose of the object based on the correspondence between the four feature points and the body coordinates of the four or eight corner points of the object. Assume that the length and width of the bounding box of the object is W and H, and the height of the object from the ground is $H_g$. We define the eight points of the bounding box of the object’s own coordinate system as:

$$\left\{\begin{array}{c}\left[-\frac{W}{2},-\frac{H}{2},\pm \frac{H_g}{2}\right],\left[\frac{W}{2},-\frac{H}{2},\pm \frac{H_g}{2}\right]\hfill \\ \left[-\frac{W}{2},\frac{H}{2},\pm \frac{H_g}{2}\right],\left[\frac{W}{2},\frac{H}{2},\pm \frac{H_g}{2}\right]\hfill \end{array}\right.$$

(6)

Since the distance from the camera to the ground object is much larger than the object’s own height $H_g$, the image coordinates of the two feature points at different heights of the object at the same latitude and longitude are very close on the image. Therefore, the corresponding relationship between the eight points of the object’s bounding box and the key points $\{P_1,P_2,P_3,P_4\}$ of the image coordinates is:

$$\left\{\begin{array}{cc}\left[-\frac{W}{2},-\frac{H}{2},\pm \frac{H_g}{2}\right]----P_1\hfill & \mathrm{left}\mathrm{top}\hfill \\ \left[\frac{W}{2},-\frac{H}{2},\pm \frac{H_g}{2}\right]----P_2\hfill & \mathrm{right}\mathrm{top}\hfill \\ \left[-\frac{W}{2},\frac{H}{2},\pm \frac{H_g}{2}\right]----P_3\hfill & \mathrm{right}\mathrm{bottom}\hfill \\ \left[\frac{W}{2},\frac{H}{2},\pm \frac{H_g}{2}\right]----P_4\hfill & \mathrm{left}\mathrm{bottom}\hfill \end{array}\right.$$

(7)

According to this correspondence, we call the PnpSolve function in OpenCV to solve the camera external parameters R and T, where R is the attitude matrix of the camera relative to the object, and T is the position of the object relative to the camera.

3.3.2. Homograph Method

We define the object as an inclined box with a width W and a length H. The origin of the object’s coordinate system is defined at the center of the box. Considering the particularity of remote sensing images, the four vertices of the object in the object coordinate system are similar on a ground plane $\pi$, and the four vertices of the object in the object coordinate system are marked as:

$$\left\{\begin{array}{c}\left[-\frac{\mathrm{W}}{2},-\frac{\mathrm{H}}{2}\right],\left[\frac{\mathrm{W}}{2},-\frac{\mathrm{H}}{2}\right]\hfill \\ \left[\frac{\mathrm{W}}{2},\frac{\mathrm{H}}{2}\right],\left[-\frac{\mathrm{W}}{2},\frac{\mathrm{H}}{2}\right]\hfill \end{array}\right.$$

(8)

According to the DOTA data format, the four points are sorted clockwise from the upper left corner. Assuming that the aspect ratio of the object is also unknown, set to $\alpha$, then the four object points can be written as:

$$\left\{\begin{array}{c}\left[-\frac{\mathrm{W}}{2},-\frac{\alpha \mathrm{W}}{2}\right],\left[\frac{\mathrm{W}}{2},-\frac{\alpha \mathrm{W}}{2}\right]\hfill \\ \left[\frac{\mathrm{W}}{2},\frac{\alpha \mathrm{W}}{2}\right],\left[-\frac{\mathrm{W}}{2},\frac{\alpha \mathrm{W}}{2}\right]\hfill \end{array}\right.$$

(9)

According to the principle of satellite remote sensing imaging, the line of sight of the imaging camera of the remote sensing image is perpendicular to the ground plane $\pi$, so the conversion of the four points on the object plane to the image plane follows the rotation transformation matrix:

$$\left[\begin{array}{c}u\hfill \\ v\hfill \end{array}\right]=\left[\begin{array}{ccc}{a}_{11}\hfill & a_{12}\hfill & a_{13}\hfill \\ -a_{12}\hfill & a_{11}\hfill & a_{23}\hfill \end{array}\right]\left[\begin{array}{c}X\hfill \\ Y\hfill \\ 1\hfill \end{array}\right]$$

(10)

Conversely, the conversion of four points on the image plane to the four points on the object plane follows the inverse of Equation (8):

$$\left[\begin{array}{c}X\hfill \\ Y\hfill \end{array}\right]=\left[\begin{array}{ccc}{a}_{11}^{\prime }\hfill & a_{12}^{\prime }\hfill & a_{13}^{\prime }\hfill \\ -a_{12}^{\prime }\hfill & a_{11}^{\prime }\hfill & a_{23}^{\prime }\hfill \end{array}\right]\left[\begin{array}{c}u\hfill \\ v\hfill \\ 1\hfill \end{array}\right]$$

(11)

According to the basic principle of affine transformation,

$$\begin{array}{cc}\hfill \left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ -a_{12}& a_{11}& a_{23}\end{array}\right]& ={\left[\begin{array}{ccc}{a}_{11}^{\prime }& a_{12}^{\prime }& a_{13}^{\prime }\\ -a_{12}^{\prime }& a_{11}^{\prime }& a_{23}^{\prime }\end{array}\right]}^{-1}\hfill \\ \hfill & =\frac{1}{{a_{11}^{\prime }}^2+{a_{12}^{\prime }}^2}\left[\begin{array}{ccc}{a}_{11}^{\prime }& -a_{12}^{\prime }& a_{12}^{\prime }{a}_{23}^{\prime }-a_{13}^{\prime }{a}_{11}^{\prime }\\ a_{12}^{\prime }& a_{11}^{\prime }& -\left(a_{11}^{\prime }{a}_{23}^{\prime }+a_{13}^{\prime }{a}_{12}^{\prime }\right)\end{array}\right].\hfill \end{array}$$

(12)

In order to solve the four parameters ${a_{11}}^{\prime }$, ${a_{12}}^{\prime }$, ${a_{13}}^{\prime }$, ${a_{23}}^{\prime }$ of affine transformation, we obtain the formulas of the four parameters and $\alpha$ according to Equation (7):

$$\left[\begin{array}{ccccc}u& v& 1& 0& 0\\ v& -u& 0& 1& -Y\end{array}\right]\left[\begin{array}{c}{a}_{11}^{\prime }\\ a_{12}^{\prime }\\ a_{13}^{\prime }\\ a_{23}^{\prime }\\ \alpha \end{array}\right]=\left[\begin{array}{c}X\\ 0\end{array}\right].$$

(13)

The four-point coordinates in Equation (9) are sequentially brought into X and Y in the above equation,

$$\left\{\begin{array}{cc}\hfill \left[X_1,Y_1\right]& =\left[-\frac{W}{2},-\frac{\alpha W}{2}\right]\hfill \\ \hfill \left[X_2,Y_2\right]& =\left[\frac{W}{2},-\frac{\alpha W}{2}\right]\hfill \\ \hfill \left[X_3,Y_3\right]& =\left[\frac{W}{2},\frac{\alpha W}{2}\right]\hfill \\ \hfill \left[X_4,Y_4\right]& =\left[-\frac{W}{2},\frac{\alpha W}{2}\right]\hfill \end{array}\right.$$

(14)

Then, we can get

$$\left[\begin{array}{ccccc}{u}_1& v_1& 1& 0& 0\\ v_1& -u_1& 0& 1& -Y_1\\ u_2& v_2& 1& 0& 0\\ v_2& -u_2& 0& 1& -Y_2\\ u_3& v_3& 1& 0& 0\\ v_3& -u_3& 0& 1& -Y_3\\ u_4& v_4& 1& 0& 0\\ v_4& -u_4& 0& 1& -Y_4\end{array}\right]\left[\begin{array}{c}{a}_{11}^{\prime }\\ a_{12}^{\prime }\\ a_{13}^{\prime }\\ a_{23}^{\prime }\\ \alpha \end{array}\right]=\left[\begin{array}{c}{X}_1\\ 0\\ X_2\\ 0\\ X_3\\ 0\\ X_4\\ 0\end{array}\right].$$

(15)

After solving Equation (14) by the least two multiplication method, it is brought into Equation (16) to get $a_{11}$, $a_{12}$, $a_{13}$, and $a_{23}$. According to the principle of perspective transformation, we can get

$$\begin{array}{cc}\hfill {\mathrm{K}}^1\mathrm{H}={\left[\begin{array}{ccc}f& 0& c_x\\ 0& f& c_y\\ 0& 0& 1\end{array}\right]}^{-1}\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{12}& a_{11}& a_{23}\\ 0& 0& 1\end{array}\right]& =\left[\begin{array}{ccc}\frac{1}{f}& 0& -\frac{c_x}{f}\\ 0& \frac{1}{f}& -\frac{c_y}{f}\\ 0& 0& 1\end{array}\right]\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{12}& a_{11}& a_{23}\\ 0& 0& 1\end{array}\right]\hfill \\ \hfill & =\left[\begin{array}{ccc}\frac{a_{11}}{f}& \frac{a_{12}}{f}& \frac{a_{13}-c_x}{f}\\ -\frac{a_{12}}{f}& \frac{a_{11}}{f}& \frac{a_{23}-c_y}{f}\\ 0& 0& 1\end{array}\right].\hfill \end{array}$$

(16)

Columns 1, 2 of $K^{-1}H$ are then unitized to obtain the matrix:

$${\left[{\mathrm{K}}^{-1}\mathrm{H}\right]}_1=\frac{f}{\sqrt{a_{11}^2+a_{12}^2}}\left[\begin{array}{ccc}\frac{a_{11}}{f}& \frac{a_{12}}{f}& \frac{a_{13}-c_x}{f}\\ -\frac{a_{12}}{f}& \frac{a_{11}}{f}& \frac{a_{23}-c_y}{f}\\ 0& 0& 1\end{array}\right].$$

(17)

The attitude matrix $R=[c_1,c_2,c_1\times c_2]$. The columns 1 and 2 of ${\left[{\mathrm{K}}^{-1}\mathrm{H}\right]}_1$ construct the columns 1, 2 $c_1,c_2$ of the matrix R. The third column of ${\left[{\mathrm{K}}^{-1}\mathrm{H}\right]}_1=$ is the offset T of the object relative to the camera.

Based on the above, we summarize the calculation process of the 3D pose for remote sensing image objects.

(1)

Predict the APP coordinates of each object through the neural network.

(2)

According to Equation (15), we can get the inverse affine transformation parameters ${a_{11}}^{\prime }$, ${a_{12}}^{\prime }$, ${a_{13}}^{\prime }$, ${a_{23}}^{\prime }$ and the object width-to-length ratio $\alpha$.

(3)

According to Equation (12), we can get the affine transformation matrix from the object coordinate system to the image coordinate system:

$$A=\left[\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ -a_{12}& a_{11}& a_{23}\end{array}\right]$$

(18)

(4)

Matrix ${\left[{\mathrm{K}}^{-1}\mathrm{H}\right]}_1$ is obtained from Equation (17).

(5)

Attitude matrix R and displacement T can be obtained by decomposing matrix ${\left[{\mathrm{K}}^{-1}\mathrm{H}\right]}_1$.

3.3.3. Object Spatial Location Using Remote Sensing Image

As shown in Figure 8, the geometric transformation of the satellite relative to the earth $\left[\begin{array}{cc}{R}_s& t_s\\ & 1\end{array}\right]$ can be obtained accurately; the rigid body connection transformation $\left[\begin{array}{cc}{R}_c& t_c\\ & 1\end{array}\right]$ of the satellite camera relative to the satellite can also be measured. Then, through the method based on APP, the transformation of the object relative to the camera can be obtained as $\left[\begin{array}{cc}{R}^T& T\\ & 1\end{array}\right]$. To sum up, it can be calculated that the 3D transform of the object relative to the earth is converted into $\left[\begin{array}{cc}{R}^T& T\\ & 1\end{array}\right]\left[\begin{array}{cc}{R}_c& t_c\\ & 1\end{array}\right]\left[\begin{array}{cc}{R}_s& t_s\\ & 1\end{array}\right]$.

4. Experiments and Analysis

4.1. Experimental Details

Experimental data. We conducted experiments on the DOTA and HRSC2016 datasets. For the DOTA dataset, we cut the image into subgraphs with a resolution of $1024\times 1024$. At training, we used batch-size = 64, and the learning rate was 0.0001 in the first 10,000 training sessions, and after every 20,000 increments, the learning rate was reduced by 0.1 times until the learning rate was equal to 0.000001. Our full convolutional network supports the input of images with different resolutions. We tested two different resolution inputs for $416\times 416$ and $1024\times 1024$. The $416\times 416$ model can get an output of $13\times 13$, $26\times 26$, $52\times 52$ scales, and the $1024\times 1024$ model can get an output of $32\times 32$, $64\times 64$, and $128\times 128$ scales, which correspond to three different object types of large, medium, and small scales, respectively. In the detecting, we first divided the DOTA image into blocks. The blocks needed to overlap to avoid cross-border loss of the object. Finally, we calculated and counted the mAP of the object. mAP is the mean Average Precision, which means the average of the AP of all object categories.

Table 4. Comparisons with the state-of-the-art methods on HRSC2016 [8].

Method	CP [15]	BL2 [15]	RC1 [15]	RC2 [15]	${\mathbf{R}}^2\mathbf{CNN}$ [31]	RRD [32]	RoI Trans. [8]	APP
mAP	55.7	69.6	75.7	75.7	79.6	84.3	86.2	86.3

and

Table 5. Comparisons with state-of-the-art detectors on DOTA. There are 15 categories, including Baseball diamond (BD), Ground track field (GTF), Small vehicle (SV), Large vehicle (LV), Tennis court (TC), Basketball court (BC), Stor- age tank (ST), Soccer-ball field (SBF), Roundabout (RA), Swimming pool (SP), and Helicopter (HC) [8].

Method	Plane	BD	Bridge	GTF	SV	LV	Ship	TC	BC	ST	SBF	RA	Harbor	SP	HC	mAP
FR-O [33]	79.42	77.13	17.7	64.05	35.3	38.02	37.16	89.41	69.64	59.28	50.3	52.91	47.89	47.4	46.3	54.13
RRPN [34]	80.94	65.75	35.34	67.44	59.92	50.91	55.81	90.67	66.92	72.39	55.06	52.23	55.14	53.35	48.22	60.67
R2CNN [35]	88.52	71.2	31.66	59.3	51.85	56.19	57.25	90.81	72.84	67.38	56.69	52.84	53.08	51.94	53.58	61.01
DPSRP [8]	81.18	77.42	35.48	70.41	56.74	50.42	53.56	89.97	79.68	76.48	61.99	59.94	53.34	64.04	47.76	63.89
RoI Trans. [8]	88.53	77.91	37.63	74.08	66.53	62.97	66.57	90.5	79.46	76.75	59.04	56.73	62.54	61.29	55.56	67.74
PnP	87.98	75.38	45.93	71.26	65.10	68.93	77.04	87.63	81.59	78.96	58.54	57.20	63.95	62.32	49.01	68.72
APP	89.06	78.23	43.52	76.39	68.42	71.62	79.05	90.42	81.51	80.51	59.48	58.91	64.21	62.19	48.46	70.13

are the statistics on the HRSC2016 and DOTA data sets, respectively. As can be seen from these two tables, the APP algorithm has the highest mAP.

We also tested five sets of models with different parameters. According to Equation (8), we could get the IoU between the predicted object point set and the true value point set, and calculate the corresponding mAP of each of these models.

Table 6. Models with different parameters. We used five groups of loss weights to get different parameter models and compared their recognition accuracy.

Number	${\mathit{\lambda }}_{\mathit{n}\mathit{o}\mathit{o}\mathit{b}\mathit{j}}$	${\mathit{\lambda }}_{\mathit{o}\mathit{b}\mathit{j}}$	${\mathit{\lambda }}_{\mathit{R}\mathit{O}\mathit{I}}$	${\mathit{\lambda }}_{\mathit{p}\mathit{t}\mathit{s}}$	${\mathit{\lambda }}_{\mathit{s}\mathit{o}\mathit{f}\mathit{t}\mathit{m}\mathit{a}\mathit{x}}$
1	1	2	0.0001	1	1
2	1	3	0.0001	1	1
3	1	5	0.0001	2	1
4	1	2	0.0001	1.5	1
5	1	2	0.1	1	1

and

Table 7. mAP for each model. It can be seen from the table that the model obtained by row 1 has the highest overall accuracy.

Method	Plane	BD	Bridge	GTF	SV	LV	Ship	TC	BC	ST	SBF	RA	Harbor	SP	HC	mAP
1	89.06	78.23	43.52	76.39	68.42	71.62	79.05	90.42	81.51	80.51	59.48	58.91	64.21	62.19	48.46	70.13
2	87.49	79.41	41.92	75.82	67.39	72.03	79.05	88.25	82.18	82.43	60.91	58.38	63.19	61.42	47.24	69.81
3	85.97	77.11	49.01	73.32	71.04	70.28	77.37	85.72	78.94	76.35	57.83	59.30	61.67	59.65	45.47	68.60
4	82.43	78.13	53.57	71.54	66.77	69.40	75.09	82.18	75.59	78.63	63.19	61.67	60.66	60.91	52.30	68.80
5	83.44	71.65	54.58	70.53	68.51	71.54	77.87	88.25	82.43	77.62	58.38	55.59	59.39	57.62	47.24	68.31

are model parameters and experimental results, respectively. By comparing the experimental results, we found that the mAP of the first set of model parameters was the highest.

The efficiency of the algorithm is one of the most important indicators for measuring the quality of the algorithm. Our model can input images in two different resolutions. As shown in

Table 8. Speed comparison with other methods (unit: ms). The LR-O, RoI Trans, and DPSRP denote RoI Transformer and the Light-Head R-CNN OBB, deformable Position Sensitive RoI pooling, respectively [8].

Image Size	LR-O	RoI Trans.	DPSRP	APP
$416\times 416$	-	-	-	13.5
$1024\times 1024$	141	170	206	140

, we used a one-stage process including NMS operation, so it was more efficient.

Table 9. Mean length of the objects.

Object	Plane	BD	Bridge	GTF	SV	LV	Ship	TC	BC	ST	SBF	RA	Harbor	SP	HC	avg
length	1.04564	1.0169	1.7242	1.4834	2.1373	3.8274	2.9355	1.9410	1.8625	1.0174	1.3142	1.0278	4.4173	1.0914	2.9527	1.9863

is the mean length of the objects. Assuming that the width W of all objects is equal to 1, the average length of each type of object also can be solved.

Experimental results. We improved yolo v3 and added four corner points of the APP prediction object based on the original region layer. The local object was predicted according to Equation (19), and then the coordinates were converted to the large image to obtain the homograph transform. Further, we decomposed the three-dimensional attitude R and displacement T of the object relative to the camera. Figure 9 is the result of the experiment. Figure 10 is an incorrectly labeled image that we can detect correctly after training.

For large images such as DOTA, we used the method of block synthesis. The large image is divided into a number of sub-blocks, each of which is just the standard size of the neural network input layer $[S_w,S_h]$, and the neural network was used to detect the object APP coordinates $(u_i,v_i)$ in each sub-block range. Assuming that the coordinate of the upper-left corner of a sub-block is $(left,top)$, the coordinate of the point in the full image is

$$\left\{\begin{array}{c}{u}_{ibig}=left+u_i\hfill \\ v_{ibig}=top+v_i\hfill \end{array}\right.$$

(19)

Converting all the objects of the sub-image to the large image, and in finally performing the NMS operation, in order to avoid the object being cut by the block, overlap between blocks was needed, and the overlap length was longer than the minimum object length. The overlap length of the horizontal and vertical sides was initially set at 20% of the basic length, as shown in Figure 11.

4.2. Error Analysis

We used the method of projecting pixel points to comprehensively evaluate the prediction accuracy of the 3D pose. The process is shown in Figure 7. When evaluating the accuracy of the 3D pose, the position and attitude of the object in three-dimensional space could be considered comprehensively by using the method of pixel projection error [36]. According to the nature of the attitude matrix R, the three rows of the matrix represent the three unit vectors of the x, y, and z axis of the camera coordinate system relative to the object, and then the three rows of R, in turn, represent the x, y, and z axis relative to the three unit vectors of the camera coordinate system. Since the y-axis of the object ontology coordinate system is the negative direction of the object orientation, the negative vector $-r_{12},-r_{22}$ of the second column of the matrix R describes the direction of the object. Therefore, the azimuth angle of the object relative to the camera coordinate system $\theta =artan\left(-r_{12},-r_{22}\right)$. The coordinate calculation Equation is

$$z_i\left[\begin{array}{c}{u}_i\\ v_i\\ 1\end{array}\right]=KR\left(X_i-T\right)i=1,2,3,4,$$

(20)

where K is the internal parameter matrix of the camera, R and T are the attitude matrix and the displacement matrix obtained according to the algorithm of Section 4.1, and $X_i$ is the coordinates of the four projection points of the object box close to the ground. The pixel error is

$$e^2={\left|\left[\begin{array}{c}{u}_i\\ v_i\\ 1\end{array}\right]-\left[\begin{array}{c}{\overline{u}}_i\\ {\overline{v}}_i\\ 1\end{array}\right]\right|}^2.$$

(21)

According to this formula, we could get the angular error and pixel error of the predicted object. We tested five different sets of parameters to get different models. It can be seen from

Table 10. Angle error and pixel error.

Number	${\mathit{\lambda }}_{\mathit{n}\mathit{o}\mathit{o}\mathit{b}\mathit{j}}$	${\mathit{\lambda }}_{\mathit{o}\mathit{b}\mathit{j}}$	${\mathit{\lambda }}_{\mathit{R}\mathit{O}\mathit{I}}$	${\mathit{\lambda }}_{\mathit{p}\mathit{t}\mathit{s}}$	${\mathit{\lambda }}_{\mathit{s}\mathit{o}\mathit{f}\mathit{t}\mathit{m}\mathit{a}\mathit{x}}$	Average Angle Error	Global Projection Pixel Error
1	1	2	0.0001	1	1	4.28	2.03
2	1	3	0.0001	1	1	4.32	2.05
3	1	5	0.0001	2	1	4.85	2.07
4	1	2	0.0001	1.5	1	4.93	2.16
5	1	2	0.1	1	1	4.78	2.37

that the error of the first set of parameters is the smallest.

5. Conclusions

In this paper, we proposed a new Anchor Points Prediction algorithm that can accurately determine the position and attitude of the object’s three-dimensional space. Differing from the traditional methods of predicting object RoI or inclined box, we used the neural network to predict multiple feature points to detect the objects. This algorithm is a one-stage algorithm, and its accuracy and efficiency have been greatly improved. It not only uniquely determines the direction of the object, but also calculates the 3D pose of the object from the APP coordinates. We believe that the APP algorithm can be better applied to object detection. Moreover, the point prediction algorithm has broad application prospects and may become a new trend in the future. Our method also has some shortcomings. For a slender object like Harbor, the bounding box is relatively large, and the object occupies only a small part of the bounding box. In this way, the features in the extracted ROI region will be inaccurate, leading to a decrease in the accuracy of predicting key points of the object.