Real-Time Registration of Unmanned Aerial Vehicle Hyperspectral Remote Sensing Images Using an Acousto-Optic Tunable Filter Spectrometer

Abstract

Differences in field of view may occur during unmanned aerial remote sensing imaging applications with acousto-optic tunable filter (AOTF) spectral imagers using zoom lenses. These differences may stem from image size deformation caused by the zoom lens, image drift caused by AOTF wavelength switching, and drone platform jitter. However, they can be addressed using hyperspectral image registration. This article proposes a new coarse-to-fine remote sensing image registration framework based on feature and optical flow theory, comparing its performance with that of existing registration algorithms using the same dataset. The proposed method increases the structure similarity index by 5.2 times, reduces the root mean square error by 3.1 times, and increases the mutual information by 1.9 times. To meet the real-time processing requirements of the AOTF spectrometer in remote sensing, a development environment using VS2023+CUDA+OPENCV was established to improve the demons registration algorithm. The registration algorithm for the central processing unit+graphics processing unit (CPU+GPU) achieved an acceleration ratio of ~30 times compared to that of a CPU alone. Finally, the real-time registration effect of spectral data during flight was verified. The proposed method demonstrates that AOTF hyperspectral imagers can be used in real-time remote sensing applications on unmanned aerial vehicles.

1. Introduction

Image registration technology refers to the process of geometric alignment of two images acquired at different times, from different perspectives, and from different sensors [1,2]. Image registration in the field of remote sensing is an essential part of many remote sensing image processes, such as object detection, urban development assessment, geographic change assessment, hyperspectral image stitching, precision agriculture, and biomedical applications [3,4,5,6,7,8]. These tasks need to be carried out to achieve successful registration; thus, further research on remote sensing image registration is deemed necessary and crucial.

Acousto-optic tunable filter (AOTF) spectral imagers are frame scanning-based imagers with an adjustable number of spectral channels, and they capture the entire data cube by sequentially exposing each band. However, in remote sensing flight experiments, changes in sensor position and attitude are caused by airflow disturbances and drone vibrations, resulting in spatial positional deviations in different spectral bands. Different spectral bands cannot be fully matched in the original data cube, leading to spectral registration errors. Therefore, it is necessary to implement post-processing registration correction.

AOTFs present advantages that include small size, light weight, no moving parts, flexible wavelength selection, and strong environmental adaptability [9], thus making them suitable as spectroscopic devices for high-resolution imaging spectrometers in aerial remote sensing [10,11]. However, in remote sensing imaging based on AOTF hyperspectral instruments, it is imperative to address any issues that may arise in the collected data.

Sharikova et al. [12] studied the spatial spectral distortion caused by acousto-optic diffraction and performed spatial and spectral calibration on imaging spectrometers based on AOTFs to develop a technique for balancing the transmittance of AOTFs throughout the entire working spectral range. To obtain undistorted data, spectral and spatial non-uniformity must be corrected. The calibration process included hardware spectral tuning of acousto-optic devices and mathematical corrections to software. The proposed method can perform real-time hardware calibration of hyperspectral devices based on AOTFs and software calibration of display results. This method is universal and suitable for other imaging spectrometers based on acousto-optic monochromaticization.

Zhang and Zhao [13] focused on the design and analysis of non-collinear AOTFs, and they used a refractive index correction to resolve the geometric parameter introduction error (0.5° or greater) that cannot be ignored in tellurium dioxide (TeO₂) non-collinear AOTFs. This basic theoretical research is crucial for the design and application of non-collinear AOTFs.

Zhao et al. [14] studied the spectral and spatial characteristics of AOTFs, such as tuning curves, spectral resolution, angular aperture, and diffraction efficiency, determined by the device’s acousto-optic crystal configuration and piezoelectric transducers. For high-throughput spectral imaging applications, it is crucial to expand the spectral bandwidth and angular aperture during the design phase of AOTFs. Therefore, this article analyzes and studies the phase mismatch caused by incident angle or wavelength using phase diagrams. In addition, a performance parameter analysis model was established for the design of large angle aperture AOTF devices based on mercuric bromide crystals, and the effects of crystal and transducer design parameters on spectral bandwidth and angle aperture were evaluated.

Yu et al. [15] investigated spectral drift, which is a unique challenge when using an AOTF spectrometer to observe moving targets, and revealed that an online spectral calibration method based on acousto-optic interaction is required. A reverse ray tracking model was constructed using the imaging position and driving frequency of the target spectrum, and it achieved real-time calibration of spectral data and ensured the stability and accuracy of subsequent target detection, recognition, and tracking. Experimental verification was conducted using the developed parallel incident light mid-infrared AOTF spectral detection system. The results showed that for simulated moving targets with different fields of view, the correction accuracy of spectral drift was greater than 4.45%. This improvement enhanced the application capability of the spectral detection of moving targets.

In our previous research on the design of an electric zoom lens AOTF spectrometer, we addressed the issue of image blurring, which causes image size deformation due to zoom, image drift due to wavelength switching, and drone platform shaking in remote sensing applications [16]. The issue of cube registration based on AOTF spectrometer imaging data has not been previously studied. According to general image processing and remote sensing image processing methods, it is widely recognized that current methods of solving remote sensing spectral data image registration mainly include grayscale and template-based, feature-based, and domain transformation-based methods, as well as machine learning and artificial intelligence-based methods [17]. Therefore, the methods for addressing remote sensing spectral data image registration can be roughly divided into these four categories.

Grayscale and template-based methods directly use correlation operations and other approaches to calculate the correlation value to identify the best matching position. Blocking matching is performed to search for sub-images similar to the template image in another image based on a known template image. Li et al. [18] proposed a deep learning semantic template matching framework for remote sensing image registration. Driven by learning-based methods, reference images and template images are taken as inputs and mapped to the semantic distribution positions of the corresponding reference images. Ruiqi et al. [19] proposed a template-matching method based on a deep global feature-based template-matching method (GFTM) to achieve fast and accurate multimodal image registration. The method performs fast template matching on global deep features to search for positions with maximum similarity. A large number of experimental results on optical and synthetic aperture radar (SAR) images have shown that the proposed method is effective for multimodal image registration.

In feature-based methods, the features of the image are extracted, feature descriptors are generated, and the features of the two images are matched based on the similarity of the descriptors. The features of an image can mainly be divided into points, lines (edges), regions (faces), and other features, as well as local features and global features. The extraction of regional (surface) features is relatively cumbersome and time-consuming; therefore, point features and edge features are mainly used. Point features include the Harris, histogram of oriented gradient (HOG), local binary pattern (LBP), scale-invariant feature transform (SIFT), speeded up robust features (SURF), binary robust independent elementary features (BRIEF), smallest univalue segment assimilating nucleus (SUSAN), features from accelerated segment test (FAST), fast retina keypoint (FREAK), binary robust invariant scalable keypoints (BRISK), oriented FAST and rotated BRIEF (ORB) algorithms and classifiers [6,7,8]. Edge features include the LoG operator, Robert operator, Sobel operator, Prewitt operator, and Canny operator. Ye et al. [20] proposed a new keypoint feature detector aimed at the simultaneous extraction of corners and spots and the calculation of SIFT descriptors for detected corners and spots and applied them jointly for remote sensing image registration. Wu et al. [21] proposed a robust and accurate feature point-matching framework. An improved SIFT method was first proposed for feature detection and matching, and it was applied to automatic remote sensing image registration. Zhang et al. [22] proposed an improved algorithm for the SURF classic algorithm, which is a short time and high-precision image registration algorithm that can meet the registration requirements of remote sensing image stitching. Chen et al. [23] proposed an iterative image registration method for remote sensing images, known as iterative scale-invariant feature transform (ISIFT). This method extends the registration system based on SIFT to a closed-feedback SIFT system that includes a correction feedback loop, iteratively updating the correction parameters. The experimental results show that compared with traditional SIFT-based methods and state-of-the-art methods, ISIFT improves performance and yields better registration accuracy. Jhan et al. [24] proposed a normalized SURF (N-SURF), which can substantially increase the number of correct matches between different multispectral image pairs, enabling one-step image registration. Additionally, they developed an automatic multispectral image registration tool suitable for multi-lens multispectral cameras. Wu et al. [25] proposed a two-step remote sensing image registration method based on local and global constraints. Experiments on multiple remote sensing image datasets have shown that this method is more robust and accurate than existing methods.

Based on domain transformation, phase correlation (Fourier Merlin transform), Walsh transform, and wavelet transform are used for registration in new domains. Ordóñez et al. [26] proposed a phase correlation algorithm based on FFT and developed a graphics processing unit (GPU) to register two remote-sensing hyperspectral images. The proposed algorithm is based on multi-layer fractional Fourier transform and logarithmic polar plots. Liu et al. [27] proposed a multi-constraint registration algorithm based on variational inference for complex remote sensing image registration problems. The experimental results showed that compared with other point set matching algorithms, their method demonstrated strong performance in terms of robustness and matching accuracy. Zhou et al. [28] proposed a novel image registration algorithm using wavelet transform and matrix multiple discrete Fourier transform, and the results showed that it can perform sub-pixel registration via full image-based methods but with shorter computation times.

Regarding machine learning and artificial intelligence-based methods, Lee et al. [29] proposed a remote sensing registration framework based on convolutional neural networks to improve the registration accuracy between two remote sensing images obtained from different times and viewpoints. The proposed high-precision registration framework was evaluated using the KOMPSAT-3 dataset and obtained a minimum root mean square error value of 34.922 based on all control points and improved the matching accuracy by 68.4% compared to traditional registration frameworks. Zeng et al. [30] proposed an image registration method based on hierarchical convolutional features and applied it to improve the efficiency of large-scale forestry image stitching generation. This method uses a deep learning architecture to adaptively obtain image features from deep convolutional neural networks. The experimental results showed that this method can detect and match image feature points with marked spectral differences and effectively extract feature points to generate accurate image registration and stitching results. Ye et al. [31] proposed a multi-scale framework with unsupervised learning called MU Net. Without expensive ground truth labels, MU Net directly learns end-to-end mapping from image pairs to their transformation parameters. The experimental results indicated that MU Net achieved more comprehensive and accurate registration between image pairs with geometric and radiative distortions. Chen et al. [32] proposed a dense connected neural network (RBDCNN) that was improved by residual blocks to extract feature values, which improved registration performance and utilized the distance difference between the transformation matrices of reference images and floating images. Compared with existing methods, the registration results were closer to that of the reference image.

With the increasing demand for high accuracy in remote sensing image registration, the complexity of remote sensing image registration algorithms is also increasing, which has led to calculation times that are too long to be used in situations with high real-time requirements. Hyperspectral image registration is a real-time application-related task, such as environmental disaster management or search and rescue scenarios [33]. The compute unified device architecture (CUDA) has advantages such as low cost, low power consumption, good portability, and flexible code modification. Therefore, by combining the advantages of a GPU in general computing and the processing speed issues faced by remote sensing image registration, a fast remote sensing image registration algorithm based on CUDA parallel computing can be developed. Liu et al. [34] utilized swarm intelligence GPUs to accelerate SAR image registration in parallel and achieve fully parallelized registration. The experimental results showed that this method can achieve approximately 40-times higher acceleration. Ordóñez et al. [35] proposed the first method of implementing hyperspectral KAZE (HSI-KAZE) with multiple nodes and GPUs for jointly registering band and multispectral images. In this method, different multispectral datasets are distributed between available nodes in the cluster using MPI, and CUDA utilizes the parallel flow-based capabilities of the GPUs within each node. Subsequently, HSI-KAZE was used in cluster systems to achieve multi-GPU registration of high-resolution multispectral images. Zhang et al. [36] proposed a multi-frame image registration algorithm and its parallel design method for high-resolution images. Compared with feature point algorithms and deep learning algorithms, the proposed algorithm and its parallel design considerably improve the registration accuracy and speed of high-resolution optical remote sensing images.

To solve the issues of image size deformation, image drift, and image jitter in the unmanned aerial vehicle remote sensing spectral data cube collected using the AOTF spectrometer based on the electric zoom lens, we used image registration methods to solve these problems. Compared with the current methods for hyperspectral image registration, the predominant approaches include grayscale and template-based, feature-based, and domain transformation-based methods, as well as some machine learning and artificial intelligence methods. A coarse-to-fine remote sensing image registration method was proposed based on feature and optical flow theory. In the coarse registration stage, the feature-based registration method was used to solve the registration problem of scale transformation, rotation, and other overall transformations. In the fine registration stage, the image registration method (based on optical flow theory) was used to solve the registration problem of the local details in the image, such as issues with image zoom produced by the use of a zoom lens and differences in the field of view caused by small jitter movements. The main objectives of this study were as follows:

• To propose a new coarse-to-fine remote sensing image registration framework based on feature and optical flow theory. The data cube composed of collected spectral segments registers the data of each spectral segment (registration between adjacent spectral segments) after resolving image blurring and spectral segment drift of the AOTF spectrometer using a fast zoom lens. The performance of the proposed method was compared with that of other advanced algorithms.
• To construct a VS2023+CUDA+OPENCV development environment for the improved demons registration algorithm based on optical flow theory, which is highly suitable for distributed and parallel processing. Parallel processing using the compute unified device architecture (CUDA) was performed to achieve rapid registration and enable real-time processing.
• To verify the proposed method based on the existing AOTF unmanned airborne spectrometer platform. The data cubes on each shooting waypoint were registered online and met the requirements of real-time registration on unmanned aerial vehicles (UAVs) and real-time processing on aircraft.

2. Related Work

This section provides information about the proposed method. First, the airborne AOTF spectrometer based on a zoom lens is introduced. Then, the image registration algorithm based on the optical flow theory is described, constituting the main part of the fine registration. Finally, the image processing algorithm acceleration based on GPU use is introduced.

2.1. Airborne AOTF Spectrometer for UAVs Based on the Zoom Lens

With the rapid development of material technology and the increasing maturity of optical device manufacturing technology, optical splitters have transformed from traditional prisms, gratings, etc., to new devices with higher spectral resolution, miniaturization, integration, and intelligence [37]. AOTF represents an all-solid-state filtering and polarization modulation device that can achieve fast electronic control tuning without mechanical moving components. This technology has many advantages, such as compact structure, high diffraction efficiency, and wide tuning range. Moreover, it has been applied in the development process of spectral imaging systems [38,39].

The airborne imaging system consists of an AOTF imaging spectrometer based on an electric zoom lens, an AOTF driver, a MINI-PC with GPU, and a battery. The composition diagram of the unmanned aerial vehicle hyperspectral imaging system based on AOTF is shown in Figure 1.

The core optical path structure of the AOTF spectrometer based on an electric zoom lens is shown in Figure 2.

In Figure 2, the core optical path structure of the AOTF spectrometer is presented, illustrating (1) electrically controlled zoom lens; (2) front objective lens; (3) aperture diagram; (4) collimating lens; (5) linear polarizer; (6) AOTF, composed of tellurium dioxide (TeO₂) crystals and piezoelectric transducers; (7) linear polarizer; (8) secondary imaging lens; (9) CMOS detector; (10) MINI-PC control and data acquisition system; and (11) RF driver.

The incident beam is refracted by an electrically controlled variable angle lens (1), a front objective lens (2), aperture diagram (3), and a collimating lens before being vertically incident on the surface of a linear polarizer (4). After polarization, the beam is vertically incident on the surface of the AOTF module. The incident light and ultrasound interact with each other inside the AOTF module to produce a diffracted beam. After passing through the linear polarizer (6), the beam is focused by a secondary imaging lens (8) on the imaging surface of the CMOS detector. Subsequently, the collected data are processed by the MINI-PC. Specifically, the polarization direction of the linear polarizer (5) is parallel to the acoustic optical interaction plane of the AOTF and perpendicular to the polarization direction of the linear polarizer (7). The purpose of using a polarizer (7) is to filter out the 0-level transmitted light.

The design materials for the AOTF spectrometer prototype based on an electric zoom lens are shown in

Table 1. Materials for the AOTF spectrometer prototype using an electric zoom lens.

Component	Parameter	Specification	Component	Parameter	Specification
AOTF filter (SGL30-V-12LE)	Wavelength	400–1000 nm	Objective lens (M112FM16)	Focal length	16 mm
	FWHM	≤8 nm		Image plane	1/1.2″
	Diffraction efficiency	≥75%		Aperture	F2.0–F16.0
	Separation angle	≥4°	Collimating lens (V5014-MP)	Focal length	50 mm
	Aperture angle	≥3.6°		Image plane	1″
	Primary deflection angle	≥2.17°		Aperture	F1.4–F16.0
	Optical aperture	12 × 12 mm	Linear polarizer (R5000490667)	Wavelength range	300–2700 nm
AOTF driver	Frequency range	43–156 MHz		Extinction ratio	>800:1
	Stability frequency	10 Hz		Size	25.4 mm
	Frequency resolution	0.1 MHz	CMOS camera (MV-CA050-20UM)	Detector	PYTHON5000
Motorized zoom lens (EL-16-40-TC-VIS-5D)	Aperture	16 mm		Pixel size	4.8 × 4.8 μm
	Response time	5 ms		Resolution	2592 × 2048
	Focal range	−10 to +10 diopters		Interface	USB 3.0

.

2.2. AOTF Spectral Characteristics

The main material of the AOTF crystal is TeO₂. Due to the characteristics of Bragg diffraction in TeO₂ crystals, the light passing through the crystal has different deflection angles for different wavelengths of diffracted light, resulting in angular displacement errors [40]. Previous studies have demonstrated the relationship between the diffraction angle and wavelength of AOTF crystals [41,42], which is described below.

The geometric relationship is satisfied by the incident light angle ${\mathsf{\theta }}_{\mathrm{i}}$, diffracted light angle ${\mathsf{\theta }}_{\mathrm{d}}$, and external diffraction angle $\mathsf{\beta }$ is defined as follows:

$$\mathit{sin}\beta =\mathit{sin}\left({\theta }_i-{\theta }_d\right)$$

(1)

where ${\mathrm{n}}_{\mathrm{d}}$ is the refractive index of TeO₂ crystals corresponding to the wavelength of the diffracted light. In addition, ${\mathrm{n}}_{\mathrm{d}}={\mathrm{n}}_{\mathrm{o}}$, where ${\mathrm{n}}_{\mathrm{o}}$ is the refractive index of $\mathrm{o}$ light when the optical rotation of the crystal is not considered. The order angle of the diffracted light can be written as follows:

$$\tan {\theta }_d={\left({\displaystyle \frac{n_o}{n_e}}\right)}^2\tan {\theta }_i$$

(2)

Equation (2) is a function of incident light wavelength. The geometric relationship is satisfied in (3) when the emitting end face of the TeO₂ crystal is not parallel to the incident end face:

$$\sin \beta =n_d\sin \left({\theta }_i-{\theta }_d-{\theta }_{\omega }\right)$$

(3)

where ${\mathsf{\theta }}_{\mathsf{\omega }}$ is the increased wedge angle of the exit end face of the crystal. Generally, the relationship between the refractive index and wavelength is shown below:

$${n_o}^2=1+{\displaystyle \frac{{2.5488\lambda }^2}{{\lambda }^2-{0.1342}^2}}+{\displaystyle \frac{{1.1557\lambda }^2}{{\lambda }^2-{0.2638}^2}}$$

(4)

$${n_e}^2=1+{\displaystyle \frac{{2.8525\lambda }^2}{{\lambda }^2-{0.1342}^2}}+{\displaystyle \frac{{1.5141\lambda }^2}{{\lambda }^2-{0.2631}^2}}$$

(5)

where ${\mathrm{n}}_{\mathrm{e}}$ is the refractive index of e-light, and both ${\mathrm{n}}_{\mathrm{o}}$ and ${\mathrm{n}}_{\mathrm{e}}$ are functions of the wavelength of light $\mathsf{\lambda }$. Based on the above equations, the expression of the external diffraction angle of the e-light crystal can be obtained as follows:

$$\beta =arcsin\left\{\begin{array}{c}{\left(1+{\displaystyle \frac{{2.5488\lambda }^2}{{\lambda }^2-{0.1342}^2}}+{\displaystyle \frac{{1.1557\lambda }^2}{{\lambda }^2-{0.2638}^2}}\right)}^{-{\displaystyle \frac{1}{2}}}\\ *\sin \left({\theta }_i-{\theta }_d-{\theta }_{\omega }\right)\end{array}\right\}$$

(6)

In this equation, the following assumptions were made: the incident light is the e-light, the incident angle ${\theta }_i\approx 22.7°$, and the wedge angle of the crystal exit end face ${\theta }_{\omega }\approx 0.6°$. The varying relationship between the diffraction angle $\beta$ and incident light wavelength is shown in Figure 2. Based on the figure, $\beta$ changes to 0.0419° over the whole wavelength range. The translation of the central image plane is approximately $f\times \tan \left(0.0419°\right)\approx 11.7 \mathsf{\mu }m$ if the focal length $f=16 mm$. We can infer that the change in the diffraction angle with respect to the wavelength over the whole wavelength range affects the imaging quality of the AOTF spectrometer.

From Figure 3, it can be seen that when the angle displacement error is small (in a small spectral range, such as 500–550 nm), the image captured on the detector focal plane will experience spectral drift. When the angle displacement error is large (wide spectral range, such as 500–900 nm), and the focal length of the imaging objective is fixed, it will cause defocusing of the scene on the detector focal plane, resulting in image blurring [16].

To address the issue of image blurring in certain spectral bands, we added a fast and tunable electric focusing lens in front of the imaging objective. The principle of an electric zoom mirror is to change its focal length through driving current, and the zoom process can be completed in milliseconds, which is in the same time order of magnitude as the time required to switch bands by changing the AOTF driving frequency. In the development process of the spectrometer, it is calibrated based on the imaging distance so that it can capture clear images in every spectral range of 400–1000 nm.

Using a fast zoom lens AOTF spectrometer design, clear images can be captured in each spectral segment; however, the size of the images will vary due to the different focal lengths of the zoom lens when capturing images in different spectral segments. To obtain aligned data cubes, registration is a crucial step that must be performed.

In addition, the registration of data cubes can also address the issue of spectral drift and the small movement of captured images caused by drone platform jitter in unmanned aerial remote sensing applications.

2.3. Image Registration Algorithm Based on Optical Flow Theory

The optical flow method is commonly used for object motion estimation in video images. It considers the inconsistency of the local motion of objects and uses the information of pixels to individually estimate their motion in a 2D space. It is a high-precision pixel-by-pixel model. The displacement field of image registration is similar to the optical flow field of moving objects, and the inconsistent spatial dislocation between remote sensing images is similar to the local motion of objects. Therefore, the optical flow method can be used for remote sensing image registration in different time phases [43].

Ideally, the object’s brightness in the image before and after the motion is assumed to be constant for small motions [44]. This assumption was followed in the proposed algorithm, rendering it similar to other optical flow algorithms. If the brightness value of point $(x,y)$ in image $I$ at time t is $I(x,y,t)$, then according to the constant brightness during movements:

$$I\left(x,y,t\right)=I\left(x+dx,y+dy,t+dt\right).$$

(7)

The Taylor expansion on the right-hand side of the above equation gives the following:

$$I(x,y,t)=I(x,y,t)+{\displaystyle \frac{\partial I}{\partial x}}\times {\displaystyle \frac{dx}{dt}}+{\displaystyle \frac{\partial I}{\partial y}}\times {\displaystyle \frac{dy}{dt}}+{\displaystyle \frac{\partial I}{\partial t}}\times {\displaystyle \frac{dt}{dt}}+\epsilon .$$

(8)

According to the prerequisite, the motion should be small, and $dx$, $dy$, and$dt$ denote small quantities; therefore, the remainder $\epsilon$ can be ignored:

$${\displaystyle \frac{\partial I}{\partial x}}\times {\displaystyle \frac{dx}{dt}}+{\displaystyle \frac{\partial I}{\partial y}}\times {\displaystyle \frac{dy}{dt}}+{\displaystyle \frac{\partial I}{\partial t}}=0.$$

(9)

Note that

$$\overrightarrow{u}=\left[\begin{array}{cc}{\displaystyle \frac{dx}{dt}}& {\displaystyle \frac{dy}{dt}}\end{array}\right], \overrightarrow{\nabla I}={\left[\begin{array}{cc}{\displaystyle \frac{\partial I}{\partial x}}& {\displaystyle \frac{\partial I}{\partial y}}\end{array}\right]}^T={\left[\begin{array}{cc}{I}_x& I_y\end{array}\right]}^T, I_t={\displaystyle \frac{\partial I}{\partial t}}.$$

(10)

Thus,

$$\overrightarrow{u}\times \overrightarrow{\nabla I}=-I_t,$$

(11)

$$\overrightarrow{u}={\displaystyle \frac{-I_t\times \overrightarrow{\nabla I}}{{\left|\overrightarrow{\nabla I}\right|}^2}}.$$

(12)

The above formulas are the core concept of the image registration algorithm based on optical flow theory. It shows that the offset of points between two images can be calculated using gradient and difference information of the image over time, i.e., the difference between the reference and the floating images. The remainder (ε) in the equation is only ignored under the condition of small motion, while ignoring a large movement will substantially impact the accuracy of the results, affecting the registration accuracy.

2.4. Basic Steps of GPU-Based Image Processing-Accelerated CUDA Program

The program can be divided into two aspects in CUDA architecture: the host side (responsible for completing complex instructions) and the device side (responsible for parallel completion of simple instructions). The host side runs on the central processing unit (CPU), whereas the device side runs on the GPU computing core. Programs running on the host side can be written in C and C++, whereas those on the device side must be built into the kernel. The general CUDA calculation process involves preparing the data to be processed on the host side, allocating storage space in the video memory, and transferring data to the video memory. The device side then performs the calculation, returns the completed results to the host side, and finally releases the video memory space [33]. The host side cannot directly manage the GPU video memory; therefore, the data transmission between the host and the device side must be realized by calling the CUDA runtime application programming interface. The data are transmitted back and forth simultaneously. As frequent back-and-forth data transmission between the host and the device is time-consuming and substantially reduces GPU execution efficiency, this type of operation should be avoided.

The host includes the CPU and host memory, and the device includes GPU and video memory. The GPU can help accelerate the CPU. The program runs on the host first and instructs the GPU to run when it encounters a device program.

The basic steps of an accelerating CUDA program with GPU image processing consist of allocating CPU memory and GPU video memory, transmitting data from the CPU to the GPU, and using the allocated grid and block to start the kernel function. After the CPU retrieves the results from the GPU, the CPU memory and GPU video memory are freed (Figure 4).

3. Methodology

This section introduces a detailed coarse-to-fine remote sensing image registration algorithm framework based on feature and optical flow theory and provides steps to implement the specific algorithm. The ORB feature point extraction and descriptor construction selected in the coarse registration stage are also introduced. Finally, the modified algorithm of demons correlation selected at the fine registration stage is introduced.

3.1. Algorithm Description

The framework of the registration algorithm is divided into two stages: a coarse registration stage based on the feature method and a fine registration stage based on optical flow theory (Figure 5).

The coarse registration stage involves the following:

• At the beginning of registration, features of the floating and reference images, which can be one or a combination of Harris, Moravec, Haar-like, HOG, LBP, SIFT, SURF, BRIEF, SUSAN, FAST, CENSUS, FREAK, BRISK, ORB, etc., are extracted.
• Extracted features are matched to obtain feature pairs of floating and reference images. This can be completed by brute force matching, which calculates the distance between a feature point descriptor and all other feature point descriptors, ranks the obtained distances, and selects the closest distance as the matching point.
• The feature alignment exception matching is deleted using the exception elimination algorithm. Common methods include using a Hamming distance of less than twice the minimum distance, cross-matching, k-nearest neighbor matching, and random sampling consistency.
• The transformation model from the floating image to the reference image is calculated using the matching feature of removing abnormal pairs.
• The floating image is transformed to match the reference image by transforming the model and adopting appropriate interpolation transformation, thus obtaining the coarse registration result.

The fine registration stage involves the following:

• The registration parameters are initialized. This may be the number of registration cycles, the similarity between the registered and reference images, or other parameters.
• It is determined whether the registration optimization conditions are met.
• If yes, the image is obtained after fine registration, and the process is completed.
• If not, the method based on optical flow theory is used to calculate the deformation displacement field.
• The deformation displacement field is subjected to Gaussian filtering.
• The filtered deformation displacement field is used to interpolate the floating image.
• The registration optimization conditions are calculated using normalization cross-correlation, mutual information, structure similarity index measure (SSIM), and difference in RMSE.
• Return to (2) and continue to judge.

The registration problems associated with scale transformation, rotation, and other global transformations are mainly addressed at the coarse registration stage. In contrast, those associated with local image details are primarily resolved at the fine registration stage. The specific algorithm is described in the pseudo-code in Algorithm 1.

Algorithm 1: A specific algorithm for coarse-to-fine remote sensing image registration based on feature and optical flow theory

Input: floating image “image01”, reference image “image02” Output: image after registration 1: Detect ORB feature point position 2: Calculate descriptor according to ORB feature point position 3: Perform feature point matching 4: Constrain the matching points to obtain excellent matching points 5: Using the matching points, calculate the projection mapping matrix from the floating image to the reference image 6: Use projection mapping matrix to complete rough image registration 7: Normalize the floating image and reference image after coarse registration and resize the square 8: Find the gradient of the reference image 9: Perform iteration operations as follows 10: Calculate coordinate offset code 11: Determine the gradient of floating image and improve the demons algorithm to determine the gradient 12: Perform Gaussian smoothing for coordinate offset to reduce burrs 13: Apply the pixel resampling code 14: Perform until convergence 15: Complete fine image registration

3.2. ORB Feature Point Extraction and Descriptor Construction

ORB is a rapid local feature detection operator proposed by Rublee et al. [45]. The ORB algorithm is an improved version of the original FAST and BRIEF algorithms [46], constructing a Gaussian pyramid and gray centroid to compensate for the scale and rotation invariances of the FAST algorithm and the rotation invariance of the BRIEF algorithm. The rotated BRIEF (RBRIEF) algorithm is used to construct the feature descriptor. The ORB algorithm has a fast-computing speed, strong real-time performance, and is insensitive to image noise.

ORB uses the FAST algorithm to detect image feature points. The FAST corner detection algorithm initially selects a central pixel (x, y) to draw a circle with a radius of 3 pixels and sets a threshold σ. The size of pixel points (x, y) is comparable to the 16-pixel values determined by the circumference in sequence. If the gray value of z pixel points on the circumference is >I (x, y) + σ or <I (x, y) − σ, then the selected pixel (x, y) is an image feature point, where I (x, y) represents the gray value of the pixel (x, y), and z is 9 or 12 [47].

It is necessary to obtain a principal direction for the feature to achieve feature points with rotation invariance. First, the feature points in the FAST corner set are taken as the center, and the gray centroid within a certain range of the feature is calculated. Then, a vector is constructed with the feature point and its gray centroid, and the main direction of the FAST feature point is obtained using the vector direction.

The neighborhood moments of feature points are defined as:

$$m_{pq}={\sum }_{x,y}{x}^p{y}^{q}I\left(x,y\right),$$

(13)

where $I\left(x,y\right)$represents the gray value of each pixel$(x,y)$ in the image: $p,q\in \left(\mathrm{0,1}\right)$.

The centroid of the neighborhood moment of the feature point is:

$$C=({\displaystyle \frac{m_{10}}{m_{00}}},{\displaystyle \frac{m_{01}}{m_{00}}}).$$

(14)

The main direction of FAST feature points is expressed as:

$$\theta =\mathrm{a}\mathrm{r}\mathrm{c}\mathrm{t}\mathrm{a}\mathrm{n} \left({\displaystyle \frac{m_{01}}{m_{10}}}\right).$$

(15)

The traditional ORB algorithm uses RBRIEF descriptors to extract binary descriptors from feature points, employing the main direction of feature points to determine the direction of binary descriptors. This addresses the issue of the lack of rotation invariance in BRIEF descriptors. The specific steps of this method are as follows:

First, a pixel block p with the size of $N\times N$ is selected after smoothing so that the gray values at the midpoint $x,y$ of the pixel block are $p\left(x\right)$ and $p\left(y\right)$. The binary test criterion $\tau$ is defined as:

$$\tau \left(p;x,y\right)=\left\{\begin{array}{c}1,p\left(x\right)<p\left(y\right)\\ 0,p\left(x\right)\ge p\left(y\right)\end{array}\right..$$

(16)

A total of n pairs of position coordinates $(x_i,y_i)$ are selected around the feature points. An n-dimensional binary feature code string is obtained by comparing the coordinates with the binary test criterion:

$$f_n(p)={\sum }_{1\le i\le n}{2}^{i-1}\tau \left(p;x_i,y_i\right).$$

(17)

A matrix $S$ of order $2\times n$ is defined as:

$$S=\left[\begin{array}{c}I,x_2,...,x_n\\ I,y_2,...,y_n\end{array}\right].$$

(18)

The principal direction of feature point $\theta$ corresponding to the rotation matrix $R_{\theta }$ is:

$$R_{\theta }=\left[\begin{array}{cc}\mathrm{c}\mathrm{o}\mathrm{s}\theta & \mathrm{s}\mathrm{i}\mathrm{n}\theta \\ -\mathrm{s}\mathrm{i}\mathrm{n}\theta & \mathrm{c}\mathrm{o}\mathrm{s}\theta \end{array}\right].$$

(19)

The matrix $\mathrm{S}$ is rotated to obtain a new matrix, ${\mathrm{S}}_{\mathsf{\theta }}$:

$$S_{\theta }=R_{\theta }S=I.$$

(20)

Finally, the binary descriptor with a rotation invariant property is obtained as follows:

$$g_n(p,\theta )=f_n\left(p\right)\left|\right(x_i,y_i)\in S_{\theta }.$$

(21)

3.3. Demons Correlation Algorithm

The demons algorithm is a registration method developed based on optical flow theory, which treats the iterative process of the algorithm as the process of each pixel in the floating image gradually spreading to the corresponding position in the reference image. The gray-value difference between the corresponding points of the two images is the external force of diffusion, whereas the gradient of the corresponding points of the reference image is the internal force of diffusion.

The offset (Ux, Uy) of point (x, y) in the reference image is calculated assuming that the reference image is S and the floating image is M, where Sx and Sy are gradients in the x and y directions, respectively, at point (x, y) in the reference image, and $\nabla f$ is the gray-value difference between the reference image and the floating image at point (x, y).

$$U_x=\nabla f\times {\displaystyle \frac{S_x}{(S_x^2+S_y^2)+\nabla f^2}},$$

(22)

$$U_y=\nabla f\times {\displaystyle \frac{S_y}{(S_x^2+S_y^2)+\nabla f^2}},$$

(23)

$$\nabla f=S(x,y)-M(x,y).$$

(24)

Sx and Sy may be solved by various methods, including a Riemann–Liouville fractional differential algorithm [48], a Grumwald–Letnikov fractional differential edge detection algorithm [49], a Prewitt gradient operator, Scharr gradient operator, and Sobel operator as the gradient algorithm. The Sobel operator was selected here based on a comparison of performances.

The coordinate offset of the whole image is calculated to make the offset smooth and continuous in the global range and is Gaussian smoothed during each iteration to avoid image blurs after resampling.

Based on the original demon algorithm, Thirion proposed to increase the diffusion velocity coefficient α to control the size of coordinate offset (diffusion velocity), as shown in Formula (25) [50]. Registration accuracy in the iterative process of the algorithm usually increases with the number of iterations α:

$$U_x=\nabla f\times {\displaystyle \frac{S_x}{(S_x^2+S_y^2)+{\alpha }^2\times \nabla f^2}},$$

(25)

$$U_y=\nabla f\times {\displaystyle \frac{S_y}{(S_x^2+S_y^2)+{\alpha }^2\times \nabla f^2}}.$$

(26)

The active demons algorithm adds the gradient of the floating image to the offset calculation [51]. The internal force driving diffusion is the gradient of the reference image in the original algorithm. The gradient of the floating image was added as the new internal force to accelerate the convergence speed of iteration. The calculation offset is shown in the following formulae, where Mx and My are the gradients in the x and y directions at point (x, y) on the floating image, respectively.

$$U_x=\nabla f\times ({\displaystyle \frac{S_x}{(S_x^2+S_y^2)+{\alpha }^2\times \nabla f^2}}+{\displaystyle \frac{M_x}{(M_x^2+M_y^2)+{\alpha }^2\times \nabla f^2}}),$$

(27)

$$U_y=\nabla f\times ({\displaystyle \frac{S_y}{(S_x^2+S_y^2)+{\alpha }^2\times \nabla f^2}}+{\displaystyle \frac{S_y}{(M_x^2+M_y^2)+{\alpha }^2\times \nabla f^2}}).$$

(28)

The inertial demons algorithm was proposed on the basis of an active demons algorithm. Here, the offset calculated by the previous iteration is added to the offset of the current iteration to further improve the convergence speed and registration accuracy [52]. Its calculation is shown in the following formula, where k is the current number of iterations, and the value of the coefficient β is between 0 and 1:

$$U_x=\beta \times U_x^{k-1}+\nabla f\times ({\displaystyle \frac{S_x}{(S_x^2+S_y^2)+{\alpha }^2\times \nabla f^2}}+{\displaystyle \frac{M_x}{(M_x^2+M_y^2)+{\alpha }^2\times \nabla f^2}}),$$

(29)

$$U_y=\beta \times U_x^{k-1}+\nabla f\times ({\displaystyle \frac{S_y}{(S_x^2+S_y^2)+{\alpha }^2\times \nabla f^2}}+{\displaystyle \frac{S_y}{(M_x^2+M_y^2)+{\alpha }^2\times \nabla f^2}}).$$

(30)

4. Experiments and Discussion

This section introduces the datasets and evaluation criteria. Then, the speed, registration rate, and robustness of 13 currently popular algorithms (ORB, KAZE, AKAZE, BRISK, optical flow method demons, B-spline-based free-form deformation algorithm, FAST, ECC, SIFT-FSC, SURF-GTM, SIFT, SURF, and VGG16) were compared in the registration of actual UAV remote sensing spectral data to evaluate the performance of the proposed method. Finally, a real-time processing experiment was conducted using UAVs for remote sensing flights.

Our algorithm runs on a 2.8–4.7 GHz Core i7-1165G7 quad-core 8-threaded processor with 16 GB RAM, an Nvidia RTX2060 independent video card, and 6 GB GDDR6 video memory. CUDA 11.4 was used under Microsoft Windows 10. The performance of the registration process of all images on the GPU and CPU was compared. The GPU implementation was written in CUDA, and the CPU version was written in standard C.

4.1. Datasets

The AOTF spectral airborne datasets 1–6 were derived from AOTF unmanned aerial remote sensing for different scenarios corresponding to roofs, towers, grassland, parking, trees, and water. The drone flew at an altitude of 100 m, capturing images in two adjacent spectral wavelengths of 580 and 620 nm. The images have a size of 2048 × 680 pixels and are shown in Figure 6.

4.2. Evaluation Criteria

SSIM [53], RMSE [54], MI [55], UIQI [56], and SAM [57] were used to evaluate the effects of different algorithms and their execution in CPU and GPU.

1.

SSIM

SSIM is an index to indicate the similarity between two images. It can measure the difference between the enhanced and real image to guide the learning process. The formula of SSIM is as follows:

$$SSIM(\mathrm{x},y)={\displaystyle \frac{(2{\mu }_x{\mu }_y+c_1)(2{\sigma }_{xy}+c_2)}{({\mu }_x^2+{\mu }_y^2+c_1)({\sigma }_x^2+{\sigma }_y^2+c_2)}},$$

(31)

where x and y are the two input images, ${\mu }_x$ and ${\mu }_y$ represent their respective averages, ${\sigma }_x^2$ and ${\sigma }_y^2$ represent their respective covariances, and $c_1$ and $c_2$ are constants with a denominator of 0. The values of SSIM range between 0 and 1, where 1 denotes two identical images. Thus, a larger value indicates that more details from the original two images have been retained.

2.

RMSE

A smaller RMSE value indicates better results, implying fewer differences between the generated image and the original two images and the retention of more details.

$$RMSE=\sqrt{{\displaystyle \frac{1}{M\times N}}{\sum }_{i=0,j=0}^{M-1,N-1}{\left[I_1(i,j)-I_2(i,j)\right]}^2},$$

(32)

where M and N are the height and width of the image, respectively, $I_1(i,j)$ is the pixel value of the reference image at position $(i,j)$, and $I_2(i,j)$ is the pixel value of the registered image at position $(i,j)$.

3.

MI

Mutual information (MI) is an important concept in information theory that describes the correlation between two systems or the amount of information they contain. In image registration, the mutual information between two images reflects the degree of mutual inclusion of information between them through their entropy and joint entropy. For images $I_1$ and $I_2$, their mutual information is represented as:

$$MI(I_1,I_2)=H\left(I_1\right)+H\left(I_2\right)-H(I_1,I_2),$$

(33)

where $H\left(I_1\right)$ and $H\left(I_2\right)$ are the entropy of images $I_1$ and $I_2$, and $H(I_1,I_2)$ is the joint entropy of images $I_1$ and $I_2$. When the similarity between two images is higher or the overlap is greater, their correlation is stronger, and the joint entropy is smaller, which means the mutual information is greater.

4.

UIQI

Universal image quality index (UIQI) is a universal objective quality evaluation index for images, for which the distortion of an image is determined by three factors: correlation distortion, brightness distortion, and contrast distortion. Although this indicator is not associated with the human visual system, its effectiveness is significantly higher than the evaluation accuracy of traditional full reference image quality objective evaluation indicators, such as the root mean square error and peak signal-to-noise ratio. Assuming $X$ is the original image, and $Y$ is the image to be evaluated, then UIQI is expressed as:

$$UIQI=\left({\displaystyle \frac{{\sigma }_{XY}}{{\sigma }_X{\sigma }_Y}}\right)\left({\displaystyle \frac{2{\mu }_X{\mu }_Y}{{\mu }_X^2+{\mu }_Y^2}}\right)\left({\displaystyle \frac{2{\sigma }_X{\sigma }_Y}{{\sigma }_X^2+{\sigma }_Y^2}}\right).$$

(34)

The range of UIQI in the formula is [−1, 1], where −1 is the worst effect, and 1 is the best effect, indicating that the image to be evaluated has no distortion; ${\mu }_X$ and ${\sigma }_X^2$ are the mean and variance of the original image pixel values, respectively; ${\mu }_Y$ and ${\sigma }_Y^2$ are the mean and variance of the pixel values of the image to be evaluated, respectively; ${\sigma }_{XY}$ is the covariance between the pixel values of the original image and the image to be evaluated.

5.

SAM

The similarity between a test spectrum (pixel spectrum) and a reference spectrum can be determined by calculating the “angle” between them. SAM can also be used to calculate the similarity between two arrays, and its calculation result can be seen as the cosine angle between the two arrays. The calculation formula is as follows:

$$SAM={\mathrm{c}\mathrm{o}\mathrm{s}}^{-1}{\displaystyle \frac{d^{T}x}{\sqrt{\left(d^{T}d\right)}\times \sqrt{\left(x^{T}x\right)}}},$$

(35)

where $d$ is the given target array, and $x$ is the array to be tested. The smaller the output value, the more similar the two arrays; the larger the distance between two arrays, the lower the similarity.

4.3. Ground Experiment Results and Analysis

FSC [58], SURF-GTM [59], SIFT [60], SURF [61], and VGG16 [3] were compared in terms of speed, registration rate, robustness, and other characteristics in the registration of actual UAV remote sensing spectral data.

The proposed registration algorithm yielded SSIM parameters of the registered and reference images that were closer to 1 for the six groups of image pairs than for the other methods, and it produced the best registration effect (

Table 2. Comparison of the structure similarity index measure (SSIM) of different registration algorithms.

Algorithms	Image Datasets
	Roofs	Tower	Grassland	Parking	Trees	Water
VGG16 [3]	0.1969	0.1058	0.1478	0.1925	0.0185	0.0967
ORB [58]	0.1740	0.1116	0.1093	0.1531	0.0550	0.0819
KAZE [59]	0.1741	0.0766	0.1787	0.1511	0.0064	0.0861
AKAZE [60]	0.1743	0.0772	0.1914	0.1505	0.0360	0.0851
BRISK [61]	0.1747	0.1053	0.1823	0.1502	0.0036	0.0797
Demons [62]	0.4500	0.4172	0.4868	0.4005	0.4006	0.3485
FFD [63]	0.1353	0.1319	0.3013	0.1279	0.1032	0.1510
FAST [64]	0.1742	0.0913	0.1825	0.1528	0.0453	0.0874
ECC [65]	0.5759	0.4210	0.5515	0.3294	0.3855	0.2110
SIFT-FSC [66]	0.1660	0.1556	0.3295	0.2227	0.1518	0.1921
SURF-GTM [67]	0.1662	0.1514	0.3343	0.2180	0.0716	0.1390
SIFT [68]	0.1724	0.0488	0.0620	0.1429	0.0637	0.0916
SURF [69]	0.1719	0.0635	0.1915	0.1507	0.0029	0.0869
Proposed	0.6661	0.7046	0.7281	0.6379	0.7557	0.5587

). The registration algorithms based on optical flow theory (demons and FFD algorithms) showed advantages over other algorithms, except in the Cars dataset. For the tested dataset, the VGG16 algorithm based on artificial intelligence and machine learning did not show substantial advantages over algorithms based only on features.

A comparison of the RMSEs of different registration algorithms (

Table 3. Comparison of root mean square error (RMSE) of different registration algorithms.

Algorithms	Image Datasets
	Roofs	Tower	Grassland	Parking	Trees	Water
VGG16 [3]	32.6225	22.0038	21.2094	26.4478	20.0595	20.3647
ORB [58]	39.8232	24.1672	21.2804	31.5082	18.5401	13.0100
KAZE [59]	39.7697	24.6450	17.0694	31.7053	18.5665	14.4373
AKAZE [60]	39.7677	24.5997	16.9560	31.6814	18.8502	13.6026
BRISK [61]	39.6348	24.2775	17.2148	31.4204	19.2785	14.5604
Demons [62]	24.3949	12.1627	8.2701	16.5976	11.6533	10.4814
FFD [63]	26.6226	19.9050	18.3493	26.7028	19.4471	19.8807
FAST [64]	39.6481	24.6021	17.0724	31.5625	16.9122	14.3712
ECC [65]	14.9568	18.9901	17.0026	25.2498	15.7109	20.8675
SIFT-FSC [66]	43.5612	23.7221	19.7786	33.0675	18.8597	22.7754
SURF-GTM [67]	43.8527	23.7213	20.0389	33.6706	20.7168	23.0848
SIFT [68]	39.3893	29.7595	22.6991	30.6268	17.0580	13.8813
SURF [69]	39.6625	25.3289	17.1490	31.5230	19.0615	13.7371
Proposed	15.6390	6.6643	4.4905	11.7514	5.2154	4.2349

) showed that the proposed algorithm yielded the smallest error among the algorithms for all datasets. VGG16 again showed no substantial advantages over other algorithms. The error of the demons algorithm was smaller than that of the FFD algorithm. With the exception of the water dataset, the feature-based ECC algorithm performed better than the other feature-based algorithms.

A comparison of the MIs of different registration algorithms (

Table 4. Comparison of mutual information (MI) of different registration algorithms.

Algorithms	Image Datasets
	Roofs	Tower	Grassland	Parking	Trees	Water
VGG16 [3]	1.1146	1.1160	0.9283	1.2110	0.3776	1.4651
ORB [58]	0.9307	1.1555	1.2037	1.1815	0.2629	1.6509
KAZE [59]	0.9566	1.1506	1.2057	1.1816	0.4022	1.7132
AKAZE [60]	0.9561	1.1509	1.2234	1.1816	0.3923	1.7153
BRISK [61]	0.9555	1.1923	1.2131	1.1801	0.3954	1.6833
Demons [62]	2.2066	2.5214	2.3533	2.2706	1.9019	2.4924
FFD [63]	1.1860	1.1899	1.1016	1.1602	0.4298	1.5313
FAST [64]	0.9548	0.7090	1.1472	1.2116	0.4442	1.6514
ECC [65]	1.9540	1.6585	1.4887	1.4983	0.8823	1.6697
SIFT-FSC [66]	0.9963	1.2388	1.1978	1.2059	0.5592	1.6172
SURF-GTM [67]	0.9886	1.2687	1.1945	1.2030	0.4320	1.5586
SIFT [68]	0.9643	0.7810	0.7534	1.1097	0.4581	1.7272
SURF [69]	0.9581	1.1029	1.2240	1.1853	0.3679	1.7154
Proposed	2.2073	2.5468	2.4975	2.3240	2.0256	2.5974

) showed that the algorithm proposed in this article has the highest mutual information value between the registered image and reference image on the selected six datasets. The demons algorithm based on the optical flow comb theory also has advantages in addition to the Cars dataset. Except for the algorithm proposed in this article and the demons algorithm, the registration results of other registration algorithms were relatively similar.

The registration effect of the proposed algorithm for eight image pairs is shown in Figure 7 in the form of image overlays. The left column of the figure shows the images to be registered superimposed on the reference images. The large difference between the images results in a small area of overlap; areas of difference are displayed in pseudo-color. The left column shows the same superposition after image registration. As the generated images are now closer to the reference images, the area of overlap is large, and the pseudo-colored area of difference is small.

In order to display the registered image and reference image, and the details after registration, a chessboard is used to alternately display the reference image, the registered image, and their details are displayed. The condition for selecting a detailed map is to select a continuous feature on the ground. If the registration is correct, the features in the detail map will be continuous. If the registration is not correct, the ground scenery will be misaligned, as shown in Figure 8 below.

The proposed algorithm performed well in all six datasets compared with the five other registration algorithms (Figure 8). The VGG16 algorithm sourced from the literature did not demonstrate noticeable performance; the feature-based SIFT-FSC and SURF-GTM algorithms yielded similar results and were not considered ideal. The optical flow theory algorithms (FFD and demons) generally performed well, but the FFD algorithm deformed images in the Cars and Roofs datasets to varying degrees after registration.

A CUDA architecture VS2023+CUDA+OPENCV development environment was built to accelerate the processing algorithm and enable the comparison of runtimes between a CPU-only and a CPU+GPU processing architecture to meet the requirements of real-time remote sensing processing using the AOTF spectrometer. The images used in this test had the same number of features (100) or the same registration accuracy. Two FFD cycles and 30 calculation cycles were used.

Even when using only two cycles, the FFD algorithm took the longest time (

Table 5. Comparison of registration time between CPU-only and CPU+GPU platforms for algorithms suitable for GPU acceleration (unit: second).

Algorithm	Platform	Image Datasets
		Roofs	Tower	Grassland	Parking	Trees	Water
FFD	CPU	2430.85	2442.26	2434.59	2439.99	2455.39	2447.01
	CPU+GPU	133.12	132.93	132.93	132.81	132.70	133.03
Demons	CPU	19.84	21.31	21.79	20.55	21.80	21.52
	CPU+GPU	0.43	0.43	0.44	0.43	0.44	0.44
FAST	CPU	1.59	2.45	7.03	3.64	2.73	3.01
	CPU+GPU	0.45	0.48	0.43	0.58	0.50	0.43
ORB	CPU	0.41	0.39	0.42	0.41	0.39	0.41
	CPU+GPU	0.34	0.34	0.34	0.35	0.34	0.34
SURF	CPU	2.02	2.70	3.37	2.02	4.27	1.57
	CPU+GPU	0.22	0.26	0.29	0.23	0.77	0.22
Proposed	CPU	15.53	16.47	16.89	15.95	16.24	16.54
	CPU+GPU	0.52	0.49	0.50	0.49	0.50	0.49

) due to the time complexity of the B-spline-based FFD algorithm of m×n×16×[c+3]×[r+3]×2 ≈ O (n⁴). Without CUDA, the computation load was substantial. The demons algorithm (based on optical flow theory) achieved a large acceleration ratio. The feature-based FAST, ORB, and SURF algorithms were relatively rapid on the CPU platform and thus did not achieve an increase in CPU+GPU. The proposed CPU+GPU algorithm achieved an acceleration ratio of ~30 times relative to that of the CPU alone. Furthermore, the absolute registration time of the two spectral segments and the feature-based algorithm remained in the same order of magnitude, and the average registration time was ~0.5 s. This indicates that the proposed method can potentially enable real-time processing of remote sensing hyperspectral data.

4.4. Real-Time Processing of the Remote Sensing Imaging Registration Experiment for Unmanned Aerial Vehicles

The waypoint acquisition procedure was modified, and the registration procedure was added to verify that the zoom lens-based AOTF spectrometer could register the collected airborne spectral imaging data cube in real time. A UAV experiment was conducted at the Xi’an Institute of Optics and Precision Mechanics, Chinese Academy of Sciences (34°10′3.12″ N, 108°51′28.67″ E) on 22 November 2023. Five flight waypoints were selected during one flight to collect five spectral data cubes. The flight altitude was 100 m, the trigger dwell time at a waypoint was 70 s, the wavelength switching step of the spectral data cube was 5 nm from 400–1000 nm, and 121 bands of data acquisition and registration were completed.

The proposed registration algorithm was adopted for image processing. Taking into account the registration effect and computation time, through multiple experiments, it was determined to set the number of extracted features to 100 and the registration iteration number to 30. A micro airborne processing platform with GPU was used to verify the effect of onboard real-time processing in terms of registration accuracy and registration time.

The registration time between two spectral bands was approximately 0.5 s (Table 5); thus, the alignment time required to complete all spectral bands was 120 × 0.5 = 60 s. The frame rate was set to 2 Hz accordingly. Since the time between waypoints was 74 s (waypoint stay of 70 s + flight time of 4 s [20 m at 5 m/s]), data acquisition of 121 spectral bands of the AOTF spectrometer and the registration of the whole data cube could be completed at each waypoint using the CPU+GPU processing mode.

The unregistered data cubes (1–5) in Figure 9 exhibited varying patterns for the three spectral segments (R: 650 nm, G: 530 nm, and B: 480 nm) of synthetic pseudo-color, suggesting that images of different spectral segments have varying fields of view for three reasons: (1) image size deformation caused by zoom, (2) image drift caused by wavelength switching, and (3) UAV platform jitter in remote sensing applications. Data cubes 6–10 in Figure 9 were registered using the proposed coarse-to-fine remote sensing image registration framework based on features and optical flow theory. Parallel processing of the CUDA architecture provided rapid registration, verifying the potential of this technique for real-time processing.

Figure 10 shows a quantitative comparison of the registration effect of waypoint data cubes and GPU registration acceleration.

Figure 10a–e shows the SSIM and RMSE between adjacent wavelengths before and after registration for the five selected waypoints in the form of curves. The SSIM between adjacent bands of the data cube after registration using the proposed algorithm was considerably greater than without registration. The RMSE of the data cube after registration was smaller than without registration.

Figure 10f compares registration time between the CPU platform and the CPU+GPU platform. Here, the registration algorithm of the five data cubes was reused offline after the flight test was completed. It can be seen from the comparative bar chart that the average processing time of adjacent spectral segments at five waypoints is approximately 15 s using CPU and 0.5 s using CPU+GPU. Therefore, the CPU+GPU processing architecture achieves approximately 30 times faster acceleration.

Finally, the hyperspectral data cube captured at the third waypoint of the experimental data was registered, and the registration performance of the algorithm proposed in this paper was compared with the algorithms in the reference literature. The comparison results are shown in

Table 6. Performance comparison of different registration algorithms for data cube registration.

Algorithm	Flight Point 3 (Cube 3)
	SSIM	RMSE	MI	UIQI	SAM
VGG16 [3]	0.0529	124.2055	0.3930	0.0580	0.6589
ORB [58]	0.2773	98.8223	1.3868	0.3415	0.5026
KAZE [59]	0.2146	108.3649	1.0637	0.2664	0.5507
AKAZE [60]	0.1898	110.5435	1.1167	0.2341	0.6031
BRISK [61]	0.1734	122.5900	0.6924	0.1950	0.5437
Demons [62]	0.3085	98.7734	1.4259	0.3802	0.5015
FFD [63]	0.0971	128.3170	0.3605	0.1071	0.5960
FAST [64]	0.1598	103.8626	1.0778	0.1955	0.5517
ECC [65]	0.1449	101.7744	1.2016	0.1806	0.5264
SIFT-FSC [66]	0.0581	119.1905	0.5133	0.0653	0.6890
SURF-GTM [67]	0.0376	124.9966	0.3564	0.0425	0.6706
SIFT [68]	0.1118	140.5192	0.1166	0.1154	0.6221
SURF [69]	0.0566	121.0245	0.4621	0.0632	0.6706
Proposed	0.3325	89.3380	1.5146	0.3998	0.4719

, which compares the performance of different algorithms in achieving data cube registration.

The registration results of different registration algorithms on data cube 3 are shown in Table 6. This article proposes a new coarse-to-fine remote sensing image registration method based on feature and optical flow theory. The registered data cubes demonstrate strong performance in terms of the SSIM, RMSE, MI, UIQI, and SAM values. Therefore, the method proposed in this article is highly appropriate for completing the registration of unmanned aerial vehicle hyperspectral remote sensing image data cubes based on AOTF spectral imager.

Currently, verification has been completed on five waypoints in a flight experiment. However, it is necessary to complete all processing steps, including concatenation of all waypoints, operation between spectral bands, and inversion of some application parameters. The processing capacity of the platform is subject to certain limitations. The processing platform with the selected GPU is currently the smallest processing platform available. If the processing platform is too large, it will be limited by the maximum takeoff weight and range of the drone. The AOTF spectral scanning imaging spectrometer has a unique advantage in selecting spectral bands. By selecting the bands of interest in the later stage and reducing the number of bands for data processing, there is potential for completing real-time hyperspectral remote sensing processing for a specific application.

5. Conclusions

A zoom lens-based AOTF spectral imager has the capability to address the issues of image blurring and spectral segment drift in remote sensing. However, image size deformation caused by zoom, image drift caused by wavelength switching, and UAV platform jitter cause slight field-of-view differences in images of adjacent spectral segments. Therefore, registration of each spectral segment is required to use these data.

A new coarse-to-fine remote sensing image registration framework based on feature and optical flow theory was proposed in this study. The issues of scale transformation, rotation, and other overall transformations were mainly addressed in the coarse registration stage, whereas issues with local image details were primarily addressed in the fine registration stage.

The proposed method was compared with current algorithms to assess its relative performance. We found the algorithm suitable for distributed and parallel processing in the fine registration stage. The algorithm’s acceleration effect was verified in a CUDA architecture (VS2023+CUDA+OPENCV) development environment.

AOTF spectral imager data were then registered in real-time during waypoint shooting, following suitable modification of the data acquisition and processing program on the unmanned aerial remote sensing platform. This proved that the proposed algorithm and the CPU+GPU platform meet the requirements of real-time registration and processing on a UAV.

Certain potential issues and limitations should be considered regarding our findings. First, it remains unknown whether increasing the number of iterations might impact the spectral energy information. Second, although the proposed algorithm achieved a relative acceleration effect of ~30 times, the absolute registration time of a group of images should be further shortened to better implement this methodology.

Several future developments are planned for the developed system. First, the effects on the spectral energy and the spectral information with an increased number of iterations when using the coarse-to-fine registration framework of the feature + optical flow method will be further investigated. Second, we will elucidate whether implementing OpenMP multithreading technology for some serial programs by the CPU and using OpenGL can improve real-time performance. Finally, research on data cube splicing will be carried out based on data cube registration.