LandNet: Combine CNN and Transformer to Learn Absolute Camera Pose for the Fixed-Wing Aircraft Approach and Landing

Abstract

Camera localization approaches often degrade in challenging environments characterized by illumination variations and significant viewpoint changes, presenting critical limitations for fixed-wing aircraft landing applications. To address these challenges, we propose LandNet—a novel absolute camera pose estimation network specifically designed for airborne scenarios. Our framework processes images from forward-looking aircraft cameras to directly predict 6-DoF camera poses, subsequently enabling aircraft pose determination through rigid transformation. As a first step, we design two encoders from Transformer and CNNs to capture complementary spatial–temporal features. Furthermore, a novel Feature Interactive Block (FIB) is employed to fully utilize spatial clues from the CNN encoder and temporal clues from the Transformer encoder. We also introduce a novel Attentional Convtrans Fusion Block (ACFB) to fuse the feature maps from encoder and transformer encoder, which can enhance the image representations to promote the accuracy of the camera pose. Finally, two Multi-Layer Perceptron (MLP) heads are applied to estimate 6-DOF of camera position and orientation, respectively. Thus the estimated position and orientation of our LandNet can be further used to acquire the pose and orientation of the aircraft through the rigid connection between the airborne camera and the aircraft. The experimental results from simulation and real flight data demonstrate the effectiveness of our proposed method.

1. Introduction

Fixed-wing aircraft landing is one of the most important and dangerous procedures during the whole flight. Therefore, it is necessary to estimate the 6-DoF (Degrees of Freedom) pose. The Global Navigation Satellite System (GNSS) and Inertial Navigation System (INS) are two types of classical measuring instruments applied to estimate the 6D pose of the aircraft.

In recent years, vision navigation research and applications have seen remarkable success. A review of the literature reveals two primary approaches based on the type of visual information used: motion estimation in unknown environments and motion estimation using known structured ground features. Motion estimation in unknown environments typically involves estimating the system’s pose by integrating the motion between consecutive images with inertial data. For example, Reference [1] extracted surrounding features on the current frame and matched them with features in the pre-built visual map. Additionally, they developed an invariant Extended Kalman Filter (EKF) for map-based visual–inertial navigation. Eduardo Gallo [2] presented a long-distance Global Navigation Satellite System (GNSS)-denied visual–inertial navigation system, which applied Visual Odometry (VO) and inertial to achieve accurate position estimation. Lee et al. [3] identified features or points of interest in the surroundings and incorporated the unknown time delays into the Visual–Inertial Odometry (VIO) system using an Extended Kalman Filter (EKF). The research in [4] applied SR-UKF [5] to fuse Direct Sparse Odometry (DSO) [6] and the inertial measurement unit to estimate the motion information of the aircraft in the landing phase. The works presented in [7,8,9] detected features in the surrounding environment and tracked them using the Lucas–Kanade method, generating keyframe-based estimates of pose changes. Their back-end systems then fused visual and inertial measurements through either filtering-based or optimization-based Visual–Inertial Odometry (VIO). Unlike the feature-based approaches mentioned above, the authors of [10,11,12] minimized a pose graph that incorporated surrounding photometric reprojection errors, IMU pre-integral errors, and relative pose errors to estimate motion, without the need for feature detection. However, these visual–inertial Simultaneous Localization and Mapping (VI-SLAM) or VIO methods operate within a local frame, making accumulation errors, or drift, inevitable in motion estimation. As a result, the navigation accuracy does not meet the requirements for the approach and landing of fixed-wing aircraft.

Motion estimation based on known structured features leverages observed characteristics with known geometric or geographic information, resulting in higher precision and drift-free pose estimation. For example, the authors of [13] presented a neural mapping method for Unmanned Aerial Vehicle (UAV) landing. They proposed a method to estimate the UAV pose based on the runway’s line features, using a flexible coarse-to-fine runway-line-detection method. Then, the UAV’s pose was acquired through the neural radiance field (NeRF). The articles [14,15] employed manually designed, structured landmarks for the correction of VIO. By leveraging the global geographic information of these landmarks, it enhanced the accuracy of navigation systems. Zhaoyang wang [16] adopted Fast R-CNN [17] to detect the runway region in the images and then calculated the position and attitude by using the detected results. Guanfeng Yu [18,19] designed a visual observation function based on runway corners with prior geographic information for visual–inertial combined navigation. The work in [20] applied optical flow and the horizon line to estimate the attitude of the fixed wing in the landing phase. Grof et al. [21] introduced an estimation method for runway-relative positioning with IMU camera data fusion. Shang et al. [22] determined the relative position/attitude parameters with the object image conjugate relationship of the runway sideline and fused the inertial information to improve the measurements.

Camera localization refers to the process of estimating the absolute camera pose (position and orientation) from an image or a sequence of images. Current image-based localization methods can be broadly categorized into three main approaches: structure-based localization, retrieval-based methods, and deep regression-based techniques:

• Structure-Based Localization Methods: These methods leverage the 3D structure of the scene and camera geometry [23]. They establish correspondences between pixels in the image and 3D scene points by matching descriptors extracted from the test image with descriptors associated with the 3D points [24,25]. The camera pose is then calculated using algorithms such as Random Sample Consensus (RANSAC) and Perspective-n-Points (PnP) [26]. However, these methods struggle with thermal images due to their low resolution and uneven noise, which significantly affects performance.
• Image Retrieval-Based Methods: These methods estimate the camera pose by matching a query image with geo-tagged images from a pre-built database. While effective in some cases, these approaches have high storage requirements and can produce inaccurate results due to the high similarity of thermal images, which makes it challenging to distinguish between different scenes.
• Deep Regression-Based Methods: These methods utilize Convolutional Neural Networks (CNNs) to learn a mapping between input images and their corresponding poses. Early work such as PoseNet [27] regressed camera poses from a single image by adding a multi-layer perceptron (MLP) head to a GoogLeNet backbone. Subsequent methods, like PoseGAN [28], improved on this by incorporating geometric structures to enhance pose estimation. MS-Trans [29] introduced transformers with a complete encoder–decoder structure to regress camera poses across multiple scenes. TransBoNet [29] employed a transformer bottleneck to estimate the absolute camera pose. MapNet [30] incorporated relative geometric constraints to improve pose estimation, though this differs from our approach in several key aspects. ALNet [31] utilized a local discrepancy perception module and an adaptive channel attention module to refine pose estimation. Yoli Shavit [32] employed a transformer encoder for camera pose regression, while MambaLoc [32] introduced a selective state-space model for visual localization.

Deep regression-based methods have also been applied to localization for flying robots. For instance, RCPNet [33] learned the relative pose for UAVs, while Baldini [34] integrated visual and inertial data to estimate UAV poses. UAVPNet [35] adopted CNNs for object detection and pose regression in UAVs.

In this paper, we propose a novel end-to-end Absolute Camera Pose Regression network named LandNet for predicting the 6-DOF camera pose in the fixed-wing aircraft landing procedure. Furthermore, the predicted camera pose can be used to compute the aircraft’s pose through a rigid connection between the airborne camera and the aircraft. The main contributions of our method are summarized as follows:

• We propose LandNet, a novel hybrid architecture combining Vision Transformer (ViT) and Convolutional Neural Networks (CNN) for 6-DoF camera relocalization using single images. LandNet is designed to efficiently generalize to large-scale environments, providing robustness for fixed-wing aircraft landing applications.
• We propose a Feature Interaction Block that fully leverages both spatial and temporal information to enhance image representations for absolute pose regression.
• An Attentional ConvTrans Fusion Block is designed to effectively integrate multi-scale and multi-level information to improve camera prediction.
• We evaluate the proposed LandNet using real flight data. The experimental results show that the method performs effectively in fixed-wing aircraft landing scenarios, especially in terms of the accuracy of the predicted orientation.

The remaining part of this paper is organized as follows: In Section 2, the preliminary knowledge of the coordinates’ definitions will be discussed. The methodology will be presented in Section 3. Section 4 will describe the experimental details, including flight data acquisition, training, and test details. Section 5 concludes the whole paper.

2. Preliminary Knowledge

2.1. Attitude Transformation

2.1.1. From Quaternion to Euler Angle

The norm of the Hamilton quaternion is defined as follows:

$$q=w+x\stackrel{\rightharpoonup }{i}+y\stackrel{\rightharpoonup }{i}+z\stackrel{\rightharpoonup }{k}$$

(1)

$$\left|\right|q\left|\right|=\sqrt{w^2+x^2+y^2+z^2}=1$$

(2)

The Euler angle can be computed from the quaternion, which is expressed as follows:

$$\left[\begin{array}{c}\theta \\ \phi \\ \xi \end{array}\right]=\left[\begin{array}{c}arctan{\displaystyle \frac{2(wx+yz)}{1-2(x^2+y^2)}}\\ arcsin\left(2\right(wy-zx\left)\right)\\ arctan{\displaystyle \frac{2(wz+xy)}{1-2(y^2+z^2)}}\end{array}\right]$$

(3)

where $\xi$, $\phi$, and $\theta$ represent roll angle, pitch angle, and yaw angle, respectively.

2.1.2. From Euler Angle to Quaternion

Given a pitch angle $\phi$, roll angle $\xi$, and yaw angle $\theta$, the quarternion is computed as follows:

$$\left[\begin{array}{c}w\\ x\\ y\\ z\end{array}\right]=\left[\begin{array}{c}cos\left({\displaystyle \frac{\phi }{2}}\right)cos\left({\displaystyle \frac{\xi }{2}}\right)cos\left({\displaystyle \frac{\theta }{2}}\right)+sin\left({\displaystyle \frac{\phi }{2}}\right)sin\left({\displaystyle \frac{\xi }{2}}\right)sin\left({\displaystyle \frac{\theta }{2}}\right)\\ sin\left({\displaystyle \frac{\phi }{2}}\right)cos\left({\displaystyle \frac{\xi }{2}}\right)cos\left({\displaystyle \frac{\theta }{2}}\right)-cos\left({\displaystyle \frac{\phi }{2}}\right)sin\left({\displaystyle \frac{\xi }{2}}\right)sin\left({\displaystyle \frac{\theta }{2}}\right)\\ cos\left({\displaystyle \frac{\phi }{2}}\right)sin\left({\displaystyle \frac{\xi }{2}}\right)cos\left({\displaystyle \frac{\theta }{2}}\right)+sin\left({\displaystyle \frac{\phi }{2}}\right)cos\left({\displaystyle \frac{\xi }{2}}\right)sin\left({\displaystyle \frac{\theta }{2}}\right)\\ cos\left({\displaystyle \frac{\phi }{2}}\right)cos\left({\displaystyle \frac{\xi }{2}}\right)sin\left({\displaystyle \frac{\theta }{2}}\right)-sin\left({\displaystyle \frac{\phi }{2}}\right)sin\left({\displaystyle \frac{\xi }{2}}\right)cos\left({\displaystyle \frac{\theta }{2}}\right)\end{array}\right]$$

(4)

2.2. Coordinates Definitions

As shown in Figure 1, the world coordinate, the World Geodetic System 1984 (WGS84) frame $\left\{\mathcal{G}\right\}$, the earth-centered, earth-fixed frame (ECEF) $\left\{\mathcal{E}\right\}$, the navigation frame (East–North–Up) $\left\{\mathcal{N}\right\}$, the body frame (Right–Forward–Down) $\left\{\mathcal{B}\right\}$, and the camera frame $\left\{\mathcal{C}\right\}$ were applied in our methodology and experiments. Furthermore, the world coordinates in this study will be presented. The transformation matrix between different frames will be presented in the following.

2.2.1. WGS84 to ECEF

Given point A in WGS84 frame, which is expressed by ${\left[\begin{array}{ccc}\mathrm{lat}& \mathrm{lon}& \mathrm{alt}\end{array}\right]}^{\mathrm{T}}$, one can obtain the position in the ECEF frame using the following equations:

$$\begin{array}{c}\left\{\begin{array}{c}{X}_{ecef}=(N+\mathrm{alt}\left)\cos \right(\mathrm{lat}\left)\cos \right(\mathrm{lon})\hfill \\ Y_{ecef}=(N+\mathrm{alt}\left)\cos \right(\mathrm{lat}\left)\sin \right(\mathrm{lon})\hfill \\ Z_{ecef}=(N{(1-f)}^2+alt)sin\left(\mathrm{lat}\right))\hfill \end{array}\right.\hfill \end{array}$$

(5)

$$N={\displaystyle \frac{a}{\sqrt{1-f(2-f)\ast {sin}^2\left(\mathrm{lat}\right)}}}$$

(6)

where lat, lon, and alt represent the latitude, longitude, and altitude of point, A respectively, a is the equatorial radius, and f is the flatness of the earth.

2.2.2. ECEF to ENU

As shown in Figure 2, the transformation from an ECEF frame to ENU frame is done with respect to the reference point. The transformation from the reference point $\left(X_r,Y_r,Z_r,\lambda ,\phi \right)$ to estimated position $\left(X_m,Y_m,Z_m\right)$ in the ENU coordinate frame is calculated as follows:

$$\left[\begin{array}{c}{X}_E\\ Y_N\\ Z_U\end{array}\right]=\left[\begin{array}{ccc}-sin\lambda & cos\lambda & 0\\ -sin\phi cos\lambda & -sin\phi sin\lambda & cos\phi \\ cos\phi cos\lambda & cos\phi sin\lambda & sin\phi \end{array}\right]\left[\begin{array}{c}{X}_m-X_r\\ Y_m-Y_r\\ Z_m-Z_r\end{array}\right]$$

(7)

where $\lambda$ is the longitude of the reference point; $\phi$ is the latitude of the reference point.

2.2.3. ENU to Body

As shown in Figure 1, the navigation frame and the body frame are defined using the East–North–Up (ENU) and Forward–Right–Up (FRU) coordinates; to make our discussion simple, we omitted the translation between two frames. The rotation transformation matrix $C_n^b$ between the ENU frame and body frame is expressed as follows:

$$C_n^b=\left[\begin{array}{ccc}cos\xi & 0& -sin\xi \\ 0& 1& 0\\ sin\xi & 0& cos\xi \end{array}\right]\left[\begin{array}{ccc}1& 0& 0\\ 0& cos\phi & sin\phi \\ 0& -sin\phi & cos\phi \end{array}\right]\left[\begin{array}{ccc}cos\theta & -sin\theta & 0\\ sin\theta & cos\theta & 0\\ 0& 0& 1\end{array}\right]$$

(8)

where $\xi$, $\phi$, and $\phi$ represent roll angle, pitch angle, and yaw angle, respectively. This transformation is shown in Figure 3.

2.2.4. Body to Camera

As shown in Figure 1, the rotation matrix $C_b^c$ from the body frame to the camera frame is calculated as follows:

$$C_b^c=\left[\begin{array}{ccc}0& 1& 0\\ 0& 0& 1\\ 1& 0& 0\end{array}\right]$$

(9)

2.2.5. World Coordinate

We establish a world coordinates system with runway point A as the origin, which is illustrated in Figure 1. Furthermore, the camera position in the ECEF frame is then transformed into a world coordinate using Equation (7).

2.3. Procedure for the Fixed-Wing Aircraft Landing

As shown in Figure 4, the landing procedure of a fixed-wing aircraft typically comprises two distinct phases: an instrument flight phase and a visual flight phase. The instrument landing phase extends from the initial approach to the Decision Altitude (DA), while the visual phase commences at DA and continues through touchdown on the runway. During the visual phase, pilots assume manual control of the aircraft for landing operations. Consequently, the provision of precise navigation information is crucial to ensure safe landing operations. The proposed pose estimation method is specifically designed to enhance navigation accuracy, particularly during the critical phase when the aircraft descends below 515 feet, where precise spatial awareness is essential for the execution of a safe landing.

3. Methodology

In this section, we present a comprehensive description of our proposed LandNet framework. The section begins with an overview of the network’s overall architecture, followed by a detailed explanations of its key components: the Initial Convolutional Blocks, CNN Encoder, Transformer Encoder, Feature Interaction Block, and Attentional ConvTrans Fusion Block. Finally, we elaborate on the design principles and implementation details of the loss function.

3.1. Network Architecture

Visual descriptors, such as local features and global representations, have long been extensively studied as counterparts [36]. Convolutional Neural Networks (CNNs) hierarchically acquire local features via consecutive convolutional operations; thus, the local cues are retained in feature maps. Transformer [37] is capable of aggregating global representations through cascaded self-attention mechanisms.

In this paper, we combine the CNNs and Transformers to take advantage of the local and global features. As illustrated in Figure 5, the proposed LandNet is composed of an Initial Convolution Blocks (ICB), two parallel branches, a Feature Interactive Block to bridge the two branches, an Attentional ConvTrans Fusion Block to aggregate two branches, and MLP layers for the regression of the pose and orientation.

3.2. Initial Convolutional Blocks

The initial convolution block comprises a 7× convolution with stride 2 followed by a 3 × 3 convolution, which is used to generate a feature map $F_{\mathrm{initial}}\in R^{64\times {\displaystyle \frac{H}{4}}\times {\displaystyle \frac{W}{4}}}$, where 64 is the channel. Then, the $F_{\mathrm{initial}}$ features are fed into two separate branches for further processing.

3.3. CNN Encoder

To obtain contextual features and certain spatial information using Convolutional Neural Networks, the CNN Encoder applies a feature pyramid architecture, where the resolution of the feature map decreases and the channel number increases with an increase in depth. As shown in Figure 6, the CNN encoder uses two types of residual structures according to whether downsampling is required.

The CNN encoder is split into four stages. Each stage is composed of multiple convolution blocks and each convolution block contains n bottlenecks [38]. Each bottleneck contains a 1 × 1 down projection convolution, a 3 × 3 spatial convolution, a 1 × 1 up-projection convolution, and a residual connection between the input and output of the bottleneck.

Let $C_1\left(\right)$ and $C_2\left(\right)$ represent the residual structures in Figure 6a and Figure 6b, respectively:

$$\begin{array}{c}{C}_1\left(x\right)=\sigma (\mathrm{BN}\left({\mathrm{W}}_2\left(\sigma \left(BN\left(W_{1}x\right)\right)\right)\right)+x)\hfill \\ C_2\left(x\right)=\sigma (\mathrm{BN}\left({\mathrm{W}}_2\left(\sigma \left(BN\left(W_{1}x\right)\right)\right)\right)+W_{s}x)\hfill \end{array}$$

(10)

where $W_1$ and $W_2$ represent the weight of the two 3 × 3 convolution operations in $C_1$ and $C_2$. $W_s$ represents the weight of the 1 × 1 convolution operation. $BN$ represents the batch normalization operation. $\sigma$ is the Rectified Linear Unit activation function (ReLU). In our implementation, we sequentially stack two residual structures to construct the CNN encoder.

The remaining stages are composed of several convolutional blocks, and each block consists of several regular consecutive 1 × 1 convolutions and 3 × 3 convolutions. The detailed information inside the CNN encoder is shown in

Table 1. Detailed information for the CNN encoder.

Stage	Kernel Size × Channels	Repeated Times
$s_1$	$\left[\begin{array}{c}1\times 1\times 64\\ 3\times 3\times 64\\ 1\times 1\times 256\end{array}\right]$	1
	$\left[\begin{array}{c}1\times 1\times 64\\ 3\times 3\times 64\\ 1\times 1\times 256\end{array}\right]$	3
	$\left[\begin{array}{c}1\times 1\times 64\\ 3\times 3\times 64\\ 1\times 1\times 256\end{array}\right]$
$s_2$	$\left[\begin{array}{c}1\times 1\times 128\\ 3\times 3\times 128\\ 1\times 1\times 512\end{array}\right]$	4
	$\left[\begin{array}{c}1\times 1\times 128\\ 3\times 3\times 128\\ 1\times 1\times 512\end{array}\right]$
$s_3$	$\left[\begin{array}{c}1\times 1\times 256\\ 3\times 3\times 256\\ 1\times 1\times 1024\end{array}\right]$	3
	$\left[\begin{array}{c}1\times 1\times 256\\ 3\times 3\times 256\\ 1\times 1\times 1024\end{array}\right]$
$s_4$	$\left[\begin{array}{c}1\times 1\times 256\\ 3\times 3\times 256\\ 1\times 1\times 1024\end{array}\right]$	1
	$\left[\begin{array}{c}1\times 1\times 256\\ 3\times 3\times 256\\ 1\times 1\times 1024\end{array}\right]$

.

3.4. Transformer Encoder

Our proposed LandNet employs the Transformer encoder to capture global context and long-range dependencies through the Multi-Head Self-Attention Mechanism (MHSA). This capability is particularly advantageous when handling significant viewpoint changes during aircraft landings. The Transformer aligns features from various image regions by assigning attention to the relevant areas, regardless of their spatial position, thereby enhancing pose accuracy.

The Transformer branch is designed following the Vision Transformer (ViT) architecture [39]. It comprises N repeated Transformer blocks, each containing two core components: a multi-head self-attention (MHSA) module and a multi-layer perceptron (MLP) block. The MLP block consists of a fully connected (fc) up-projection (fc) layer followed by a fc down-projection layer. To ensure stable training and improved performance, Layer Normalization (LayerNorm) is applied before each layer, and residual connections are incorporated in both the self-attention layer and the MLP block.

For tokenization, the feature maps produced by the initial convolutional module are compressed into non-overlapping 14 × 14 patch embeddings through a linear projection layer. This projection is implemented as a 4 × 4 convolutional layer with a stride of 4, effectively transforming the input feature maps into a sequence of patch tokens.

The scaled dot-product attention mechanism maps a query matrix $\mathbf{Q}$, key matrix $\mathbf{K}$, and value matrix $\mathbf{V}$ to an attention matrix. A scaling factor is used to prevent very small gradients.

The scaled dot-product attention can be expressed as follows:

$$Attention(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathrm{softmax}\left({\displaystyle \frac{\mathbf{Q}{\mathbf{K}}^T}{\sqrt{C}}}\right)\mathbf{V}$$

(11)

The multi-head attention joint obtains information from diverse subspaces, providing the concatenation of different heads to improve the modelling capability:

$$\begin{array}{c}\mathrm{MultiHead}(\mathbf{Q},\mathbf{K},\mathbf{V})=Concat(hea{d}_1,hea{d}_2,\dots ,hea{d}_h){\mathbf{W}}^{\mathbf{O}}\hfill \end{array}$$

(12)

where $hea{d}_i=\mathrm{Attention}({\mathbf{QW}}_i^Q,{\mathbf{KW}}_i^K,{\mathbf{VW}}_i^V)$ is the projected matrix of the output. ${\mathbf{W}}_i^Q$, ${\mathbf{W}}_i^K$, and ${\mathbf{W}}_i^V$ are the corresponding parameter matrices, respectively.

The Transformer branch is built by stacking multiple Transformer encoders. Given the embedded feature $z_0$, the Transformer encoder structure with L layers can be described as follows:

$$\begin{array}{c}{z^{\prime }}_l=\mathrm{MHSA}\left(\mathrm{LN}\left(z_{l-1}\right)\right)+z_{l-1},l=1,2,\dots ,L\hfill \\ z_l=\mathrm{MLP}\left(\mathrm{LN}\left(z_l\right)\right)+{z^{\prime }}_l,l=1,2,\dots ,L\hfill \end{array}$$

(13)

where LN() is the layer normalization. By constructing a Transformer encoder with L identical layers, the encoder output, $z_l\in R^{N\times C}$ has the same size as the input $z_0\in R^{N\times C}$. The Transformer encoder is illustrated in Figure 7.

3.5. Feature Interactive Block

The feature maps $F^{C\times H\times W}$( $C,H,W$ represent channel, height, and width, respectively) were obtained from the CNN encoder and patch embeddings $P^{K\times E}$ were obtained from the Transformer branch, where K and E represent the number of image patches and the dimensions of embedding. Then, a Feature Interactive Block (FIB) was designed to eliminate the misalignment between the two feature maps.

When the feature map $F^{C\times H\times W}$ is fed to the Transformer branch, our FIB module first adopts a downsampling operation to finish the spatial dimension alignment. After that, feature concatenation is performed to fuse the local features and global features.

On the other hand, when the $P^{K\times E}$ from the Transformer is fed into the CNN branch, our FIB performs an upsampling operation to the patch embeddings to align them with the spatial size of the feature map.

Furthermore, a 1 × 1 regular convolution is applied to adjust the channel dimension. After the channel dimension and spatial dimension are aligned, an element-wise addition is employed to complete the feature enhancement for the CNN encoder. An illustration of the FIB is shown in Figure 8.

3.6. Attentional ConvTrans Fusion Block

Element-wise addition is the most simple way to fuse the output of the Transformer branch to the output of the CNN branch Transformer encoder. However, this fusion method omits the relationship inside the feature maps, which leads to a bad fusion effect. To solve this problem, motivated by Peng [40], an Attentional Convtrans Fusion Block (ACFB) was developed to fuse the feature maps from the two branches to improve the feature quality. The structure of the proposed ACFB block is shown in Figure 9.

The procedure of ACFB block is expressed as follows:

$$F_{out}=Concat(F_{CNN}\otimes f_s,F_{Transformer}\otimes f_c)$$

(14)

where $F_{CNN}$ and $F_{Transformer}$ are the CNN feature map and Transformer feature map, respectively, $f_s$ is the spatial attention module, and $f_c$ is the channel attention module. ⊗ refers to matrix multiplication.

Let $f_{conv}$ represent the convolution operation. $f_{mean}$ and $f_{max}$ are the mean operation and maximum operation, respectively. The spatial attention map in our ACFB module is computed as follows:

$$Spatial=\sigma \left(f_{conv}Concat(f_{mean},f_{max})\right)$$

(15)

Following SENet [41], the channel attention module is used to refine the Transformer feature map.

3.7. Loss Function Design

Our camera localization method is optimized to minimize the deviation between the ground truth $({\mathbf{x}}_{truth},{\mathbf{q}}_{truth})$ and the predicted position and pose $({\widehat{\mathbf{x}}}_{predicted},{\widehat{\mathbf{q}}}_{predicted})$. Euclidean distance is used to evaluate the position loss $L_x$ and the attitude loss $L_q$, which are calculated as follows:

$$L_x=\left|\right|{\mathbf{x}}_{truth}-{\widehat{\mathbf{x}}}_{predicted}|{|}_2$$

(16)

where $\mathbf{q}$ is normalized to a unit vector to ensure that it can be transformed into a valid rotation matrix. ${\mathbf{x}}_{truth}$ and ${\widehat{\mathbf{x}}}_{predicted}$ are both transformed into the ENU frame. Considering the scale difference between the two types of losses, two learned parameters $s_x$ and $s_q$ are used to balance the two losses. The loss function then is defined as follows:

$$\mathcal{L}=L_{x}exp(-s_x)+s_x+L_{q}exp(-s_q)+s_q$$

(17)

where $\mathcal{L}$ represents the overall loss.

4. Experiments

4.1. Datasets

4.1.1. Simulation Data

The landing procedure of a fixed-wing UAV equipped with a forward-looking camera was simulated using an Unreal Engine. The coordinate system definition during the simulation was consistent with that in Section 2.2. The Unreal Engine simulation of the aircraft landing scene is shown in Figure 10. The simulated images and corresponding camera poses were used for training and validation of the algorithm.

4.1.2. Flight Data Acquisition

As shown in Figure 11, our flight platform consists of an FLIR camera, an Inertial Measurement Unit (IMU), a recorder for flight-related data, and a Differential Global Positioning System (DGPS). The IMU and recorder were installed in the cabin, which is considered the aircraft’s center of gravity. Prior to flight, the rotation matrix and translation vector between the FLIR camera and IMU were calibrated. The DGPS provides high-accuracy position and orientation data, which served as the ground truth for our method. The calibration data between the camera and IMU are presented in

Table 2. The calibrated parameters of the IMU and FLIR cameras.

Parameters Type	Specific Parameters	Values
Camera intrinsic parameters	Pixel size	$0.025\mathsf{\mu }$m
	Resolution	640 × 512
	Focal length	$\left\{\begin{array}{c}{f}_x=1010.7\mathrm{pixel}\hfill \\ f_y=1009.5\mathrm{pixel}\hfill \end{array}\right.$
	Radial distortion	$\left\{\begin{array}{c}{k}_1=-0.3408\hfill \\ k_2=0.1238\hfill \end{array}\right.$
IMU installation on the aircraft	Position	$\left[\begin{array}{ccc}0.0704& -0.4742& -7.2863\end{array}\right]$ m
Camera installation	Attitude	$\left[\begin{array}{ccc}-1.101882& -5.366247& -0.070693\end{array}\right]deg$
	Position	$\left[\begin{array}{ccc}-0.007& 0.190& -12.229\end{array}\right]$ m

. The data-collection flight test was conducted with an infrared camera onboard at Pucheng Airport, Shaanxi Province, China, and the weather conditions were foggy. A total of three flight sessions were carried out, collecting images below a flight altitude of 515 feet, with 1404 images taken per session. The distribution of images according to flight altitude is as follows: 27% of the images were taken at altitudes between 515 and 400 feet, 20% at altitudes between 400 and 300 feet, 20% at altitudes between 300 and 200 feet, and 33% at altitudes below 200 feet. Images captured by the airborne FLIR camera are shown in Figure 12.

4.2. Data Preparation

It is necessary to transform the aircraft’s attitude information into the camera’s attitude. The camera can be obtained from the aircraft’s pose using Equations (5)–(9) through a rigid connection. After that, we split all the images and corresponding pose data into a training set and a test set.

Data Processing

We begin by processing the position data within a world coordinate system, applying normalization to scale the data to a standardized range.

In addition, the images captured by the airborne forward-looking camera undergo a series of transformations. First, rescaling is performed to ensure uniformity in the size of all training and test images, which is essential to ensure they are compatible with the CNN inputs. To further enhance performance in large-view scenarios for the LandNet model, data augmentation techniques such as random rotation and random cropping are applied.

Moreover, during the training phase, adjustments to the brightness, contrast, and saturation of input images are made. These modifications increase the variability of the dataset, enabling LandNet to train on images with diverse visual conditions, thereby improving its robustness.

4.3. Training Details

All the training and test experiments are performed on Pytorch ver. 2.6 [42] platform on a single NVIDIA RTX3090 GPU card with Windows 11, where we set the batch size to 16, the learning rate to $1\times {10}^{-3}$, the weight decay to $5\times {10}^{-4}$, and the epoch to 300.

4.4. Testing Metric Definitions

The accuracy of position and orientation estimations were used to evaluate the effectiveness of the proposed method in our experiments. To be specific, we compared the specific aircraft position and orientation errors in the navigation frame (East–North–Up) with the differential DGPS values.

4.5. Experimental Results

4.5.1. Simulation Results

The simulated images and corresponding camera poses were used for training and validation of the algorithm. PoseNet [27], IRPNet [43], Atloc [44], HyperPoseNet [45], and TransPose [46] were used to compare the accuracy of position and orientation errors with our LandNet below a flight altitude of 500 feet using the Root Mean Square Error (RMSE). The experimental results are reported in

Table 3. The accuracy of the simulated datasets.

Method	$\mathbf{\Delta }\mathit{X}$ (m)	$\mathbf{\Delta }\mathit{Y}$ (m)	$\mathbf{\Delta }\mathit{Z}$ (m)	$\mathbf{\Delta }\mathit{\xi }$ (deg)	$\mathbf{\Delta }\mathit{\phi }$ (deg)	$\mathbf{\Delta }\mathit{\theta }$ (deg)
PoseNet [27]	13.9	9.2	1.3	0.235	0.381	0.328
IRPNet [43]	11.1	10.2	1.5	0.286	0.394	0.271
Atlocc [44]	10.8	8.5	1.4	0.301	0.372	0.249
HyperPoseNet [45]	8.9	9.1	1.19	0.201	0.453	0.232
TransPose [46]	8.4	8.7	1.26	0.257	0.371	0.295
LandNet	7.6	7.3	0.9	0.110	0.339	0.218

.

$\mathsf{\Delta }X$, $\mathsf{\Delta }Y$, and $\mathsf{\Delta }Z$ represent the differences between the predicted values and the ground truth. $\mathsf{\Delta }\xi$ (deg), $\mathsf{\Delta }\phi$, and $\mathsf{\Delta }\theta$ refer to the differences in the Euler angles between the estimated values and the ground truth. Table 3 shows that our proposed LandNet achieved the smallest errors in the X, Y, and Z positions (7.69 m, 7.36 m, and 0.96 m, respectively). This indicates that LandNet performs better in accurately predicting the 3D coordinates compared to the other methods listed.

In terms of orientation, LandNet has a relatively low angular error ($0.{218}^{\circ }$), which suggests that it offers better rotation accuracy, particularly when compared to models like PoseNet ($0.{328}^{\circ }$) and TransPose ($0.{295}^{\circ }$).

Additionally, our proposed LandNet offers a well-balanced performance across all metrics (position and orientation). While models like HyperPoseNet and TransPose achieve a slightly better performance in specific aspects (e.g., PoseNet in orientation), LandNet consistently obtained fewer positioning and orientation errors, which may suggest it has a more stable or reliable overall performance.

4.5.2. Results on Real-Flight Data

We conducted an experimental comparison of our approach with contemporary localization schemes, which are categorized into two classes: (i) CNN-based Camera Pose Regression methods, including PoseNet [27], IRPNet [43], Atloc [44], and HyperPoseNet [45]. (ii) Transformer-based Camera Pose Regression methods, including SitPose [47], Transposenet [46], Transecondernet [48], and our LandNet. The experimental results are presented in

Table 4. The comparison results on flight data.

Flight Height (ft)	Category	Methods	$\mathbf{\Delta }\mathit{X}$ (m)	$\mathbf{\Delta }\mathit{Y}$ (m)	$\mathbf{\Delta }\mathit{Z}$ (m)	$\mathbf{\Delta }\mathit{\xi }$ (deg)	$\mathbf{\Delta }\mathit{\phi }$ (deg)	$\mathbf{\Delta }\mathit{\theta }$ (deg)
515–400	CNN-based	PoseNet [27]	12.5	5.7	0.9	0.070	0.080	0.020
		IRPNet [43]	12.3	10.5	1.9	0.018	0.009	0.038
		Atloc [44]	9.5	7.9	1.1	0.010	0.033	0.025
		HyperPoseNet [45]	9.2	5.8	1.8	0.020	0.010	0.030
	Transformer-based	SitPose [47]	12.8	35.7	3.34	0.058	0.060	0.203
		TransPose [46]	9.4	8.7	1.6	0.053	0.048	0.051
		TransencoderNet [48]	8.3	9.3	1.1	0.045	0.034	0.029
		LandNet	6.2	3.5	0.2	0.020	0.010	0.030
400–300	CNN-based	PoseNet [27]	6.5	3.9	1.1	0.002	0.020	0.050
		IRPNet [43]	13.6	6.4	0.9	0.021	0.004	0.029
		Atloc [44]	7.5	5.2	2.6	0.004	0.006	0.008
		HyperPoseNet [45]	11.1	5.9	1.2	0.006	0.003	0.008
	Transformer-based	SitPose [47]	10.0	7.5	0.4	0.009	0.033	0.104
		TransPose [46]	9.3	5.5	1.2	0.036	0.048	0.029
		TransencoderNet [48]	8.8	6.6	0.9	0.032	0.031	0.032
		LandNet	7.0	3.7	0.7	0.003	0.002	0.007
300–200	CNN-based	PoseNet [27]	9.2	4.4	1.9	0.003	0.020	0.003
		IRPNet [43]	22.2	10.6	0.88	0.016	0.001	0.014
		Atloc [44]	11.8	7.2	2.5	0.004	0.001	0.006
		HyperPoseNet [45]	7.3	3.8	1.7	0.006	0.004	0.040
	Transformer-based	SitPose [47]	13.5	8.5	1.1	0.025	0.016	0.073
		TransPose [46]	9.5	5.3	1.3	0.021	0.030	0.015
		TransencoderNet [48]	8.3	5.6	0.9	0.022	0.027	0.019
		LandNet	7.5	2.8	0.9	0.003	0.001	0.006
Below 200	CNN-based	PoseNet [27]	14.5	6.0	1.4	0.015	0.017	0.030
		IRPNet [43]	9.2	5.8	1.8	0.021	0.007	0.033
		Atloc [44]	5.1	4.9	1.8	0.005	0.001	0.009
		HyperPoseNet [45]	15.0	3.9	2.2	0.017	0.002	0.030
	Transformer-based	SitPose [47]	12.5	11.4	1.2	0.015	0.003	0.015
		TransPose [46]	9.3	6.0	1.1	0.021	0.033	0.018
		TransencoderNet [48]	6.3	7.3	0.8	0.020	0.019	0.021
		LandNet	4.9	6.5	0.9	0.016	0.002	0.001

.

$\mathsf{\Delta }X$, $\mathsf{\Delta }y$, and $\mathsf{\Delta }Z$ represent the position estimation errors in the world frame along the respective axes. Similarly, $\mathsf{\Delta }\xi$, $\mathsf{\Delta }\phi$, and $\mathsf{\Delta }\theta$ denote the orientation estimation errors of the camera’s roll angle, pitch angle, and yaw angle, respectively. These orientation errors are derived from the quaternion output of the proposed LandNet model.

As shown in Table 4, our proposed LandNet employs a hybrid architecture that combines CNNs and Transformers, effectively leveraging the strengths of both models. This design enables LandNet to achieve s competitive performance across varying flight altitudes. For instance, LandNet demonstrates exceptional accuracy in the Z-direction, achieving an error of just 0.2 m within the 515-400 ft flight height range-significantly outperforming other methods in the same conditions.

To be specific, PoseNet has a $\mathsf{\Delta }Z$ of 0.9 m, IRPNet shows a 1.9 m error of $\mathsf{\Delta }Z$, and Atloc has a $\mathsf{\Delta }Z$ of 1.1 m. The relatively low $\mathsf{\Delta }Z$ of 0.2 m from LandNet demonstrates its superior navigation capabilities for fixed-wing landings.

While LandNet’s performance in the X and Y directions (errors of 6.2 m and 3.5 m, respectively) is not as outstanding as that in the Z-direction, it still performs well compared to other models in the 515–400 ft range. For example, PoseNet shows errors of 12.5 and 5.7, and IRPNet has errors of 12.3 and 10.5 in the X and Y directions. LandNet’s errors are much lower, especially in the X direction. TransPose has error of 11.3 m. TransencoderNet shows errors of 8.8 m. Also, LandNet consistently achieves the smallest $\mathsf{\Delta }X$, $\mathsf{\Delta }Y$, and $\mathsf{\Delta }Z$ errors across all height ranges.

LandNet also performs well in minimizing Euler angle errors. In the 515–400 ft range, the Euler angle errors are $0.020^{\circ }$, $0.010^{\circ }$, and $0.030^{\circ }$, showing that it outperforms many other methods in terms of orientation accuracy. At lower heights (below 200 ft), LandNet achieves Euler angle errors of $0.016^{\circ }$, $0.002^{\circ }$, and $0.001^{\circ }$, which is significantly better than the other methods.

Compared to other CNN-based and Transformer-based methods like PoseNet, IRPNet, and TransPose. LandNet provides lower $\mathsf{\Delta }X$, $\mathsf{\Delta }Y$, $\mathsf{\Delta }Z$, and Euler angle errors. Even transformer-based methods, like SitPose and TransencoderNet, which perform well in specific ranges, cannot match LandNet’s overall accuracy.

In particular, SitPose exhibited significant errors in the real-flight data. This may be due to its nature as a relative camera pose regression method, where it learns the motion between adjacent image frames. This approach can lead to substantial errors, especially in landing scenarios.

In conclusion, LandNet is best-suited for the landing phase of the fixed-wing aircraft, which demands high precision in vertical measurements.

We also provide a visual comparison with PoseNet, HyperposeNet, TransPoseNet, and our proposed LandNet. The visual comparison results are shown in Figure 13 and Figure 14.

From Figure 13, it can be observed that the proposed LandNet achieved the most optimal position estimation throughout the entire aircraft landing process compared to PoseNet, HyperPoseNet, TransPoseNet, and differential GPS (ground truth). To further evaluate LandNet’ performance, we divided the flight phases into four stages—515–400 ft, 400–300 ft, 300–200 ft, and below 200 ft—as illustrated in Figure 14.

As shown in Figure 14, during the 515–400 ft altitude phase, TransPoseNet exhibited significant estimation errors. This may stem from its exclusive reliance on the Transformer architecture, which lacks the ability to capture local spatial features. In the 400–300 ft phase, both PoseNet and HyperPoseNet demonstrate a degraded performance. This is likely due to the aircraft’s sharp turn to align with the runway during this stage, which induces substantial viewpoint changes that challenge the robustness of CNN-based methods in handling large perspective variations.

In the 300–200 ft phase, all baseline methods show noticeable performance deterioration, whereas the proposed LandNet makes relatively accurate position estimations. This highlights LandNet’s enhanced robustness due to its hybrid design that synergistically combines the strengths of CNNs (local feature extraction) and Transformers (global context modeling). These results validate the effectiveness of LandNet in addressing complex dynamic scenarios during critical landing phases.

4.6. Ablation Study

To assess the effectiveness of the proposed module in LandNet, we conducted a series of ablation studies. We compared the performance of each module against the ground truth in terms of RMSE.

4.6.1. Ablation on Parameter Inside CNN and the Transformer

First, we investigated the parameters within both the CNN and Transformer branches. For the CNN branch, we adjusted the number of channels in the first stage. For the Transformer branch, we explored different numbers of embedding dimensions and attention heads. The experimental setup and corresponding results are presented in

Table 5. Ablation setting and parameter results.

Transformer Branch	CNN Branch	$\mathbf{\Delta }\mathit{X}$ (m)	$\mathbf{\Delta }\mathit{Y}$ (m)	$\mathbf{\Delta }\mathit{Z}$ (m)	$\mathbf{\Delta }\mathit{\xi }$ (deg)	$\mathbf{\Delta }\mathit{\phi }$ (deg)	$\mathbf{\Delta }\mathit{\theta }$ (deg)
Embedding Dimension	Heads	Channel
384	6	64	12.34	9.12	1.91	0.035	0.025	0.063
		128	11.08	8.52	1.58	0.032	0.016	0.040
		512	10.98	7.92	1.12	0.026	0.013	0.032
576	9	256	10.65	8.31	1.56	0.023	0.011	0.036
768	12	256	9.34	6.07	0.93	0.020	0.005	0.031

.

We can observe that the accuracy improves as we increase the parameters in either the CNN or Transformer branch. Increasing the parameters in the CNN branch yields greater improvements, while also increasing the computational overhead costs.

4.6.2. Abaltion Study on FIB and ACFB Module

As shown in

Table 6. Ablation studies results on FIB and ACFB.

Module		$\mathbf{\Delta }\mathit{X}$ (m)	$\mathbf{\Delta }\mathit{Y}$ (m)	$\mathbf{\Delta }\mathit{Z}$ (m)	$\mathbf{\Delta }\mathit{\xi }$ (deg)	$\mathbf{\Delta }\mathit{\phi }$ (deg)	$\mathbf{\Delta }\mathit{\theta }$ (deg)
FIB	ACFB
✓	✓	9.34	6.07	0.93	0.020	0.005	0.031
✓		10.45	6.78	1.22	0.026	0.006	0.030
	✓	9.56	7.12	1.32	0.030	0.011	0.038

, we calculated the accuracy of orientation and position using RMSE for flight altitudes below 515 feet. Our results indicate that the proposed ACFB module positively impacts both orientation and position accuracy, yielding an improvement of approximately 1 meter in position.

4.6.3. Sensitivity Study for $s_x$ and $s_x$

The hyperparameters $s_x$ and $s_q$ are learned parameters that control the balance between the two losses. In the camera pose regression task, quaternions are commonly used to represent orientation due to their continuous differentiability. Since we expect quaternion values to be much smaller in magnitude, their noise should be significantly lower than the position noise. Therefore, $s_q$ should be much smaller than $s_x$, meaning that the orientation loss should be weighted much higher than the position loss.

To investigate how performance varies with changes in the hyperparameters $s_x$ and $s_q$, we conducted a sensitivity study. Specifically, we selected values for $s_x$ and $s_q$ from predefined ranges, trained the model with these hyperparameters on the training set, and reported the pose prediction error on the test set. The experimental settings and results are summarized in

Table 7. Experimental results for the $s_x$ and $s_q$.

${\mathit{s}}_{\mathit{x}}$	${\mathit{s}}_{\mathit{q}}$	$\mathbf{\Delta }\mathit{X}$ (m)	$\mathbf{\Delta }\mathit{Y}$ (m)	$\mathbf{\Delta }\mathit{Z}$ (m)	$\mathbf{\Delta }\mathit{\xi }$ (deg)	$\mathbf{\Delta }\mathit{\phi }$ (deg)	$\mathbf{\Delta }\mathit{\theta }$ (deg)
3	0	9.34	6.07	0.93	0.020	0.005	0.031
0	0	14.93	7.74	1.29	0.024	0.006	0.033
0	2	12.20	7.62	0.91	0.022	0.006	0.03

. Additionally, we compared the accuracy of position and orientation errors below a flight altitude of 515 feet using the Root Mean Square Error (RMSE).

As shown in Table 7, the three groups of $s_x$ and $s_q$ exhibited a similar performance in terms of orientation accuracy. However, higher values of $s_q$ can negatively impact position accuracy. Therefore, we set $s_x$ = 3 and $s_q$ = 0 as the initial values for our LandNet.

5. Conclusions

In this paper, we presented a novel absolute camera pose estimation network called LandNet for fixed-wing aircraft landings. Our proposed LandNet utilizes images from the forward-looking camera to estimate the 6-DOF camera pose.

We first introduced the basic concepts related to the landing procedures of fixed-wing aircraft, including coordinate definitions and the overall landing process. Following this, we presented the architecture of our hybrid network, which combines CNN and Transformer models. We then discussed the Feature Interactive Method, which enhances the mutual feature representation capabilities of both networks. An Attentional ConvTrans block was introduced to fuse the feature maps from the CNN and Transformer. Finally, two MLP pose regressors were employed to estimate the camera’s position and orientation. As a result, the fixed-wing aircraft’s attitude and position can be derived through rigid connection using pre-calibrated parameters. Experimental results using simulation data and real flight data demonstrated the effectiveness of the proposed method. However, the estimated position error in the eastward and northward directions remains significant. To address this issue, we plan to implement the following measures in the future:

Enhanced Data Augmentation: To handle image feature changes across altitude ranges, introduce more diverse augmentation techniques such as varying lighting conditions, simulating different textures, or introducing perspective shifts. This would help the model better adapt to various conditions during training.

Improved Data Collection and Preprocessing: To address data quality issues, improve the data collection process to ensure that the dataset represents a wide range of altitudes and conditions. Additionally, perform data cleaning to remove noise and ensure that all data are consistent and accurate.

Model Architecture Refinement: Deeper layers, the Adaptive Attention Mechanism, and fine-tuning operations should be considered in the future to improve the accuracy of LandNet.